2020-10-15T16:16:11.214 INFO:root:teuthology version: 1.0.0-b106d09 2020-10-15T16:16:11.216 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --owner scheduled_yuvalif@teuthology --lock --block --archive /home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098 --name yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi --verbose -- /tmp/teuthology-worker.arxkucje.tmp 2020-10-15T16:16:11.245 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-10-15T16:16:11.328 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098 branch: wip-yuval-cls-fifo description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: null first_in_suite: false job_id: '5527098' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-yuval-cls-fifo ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-yuval-cls-fifo sha1: 417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 owner: scheduled_yuvalif@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuval-cls-fifo suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.15069 2020-10-15T16:16:11.329 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa; will attempt to use it 2020-10-15T16:16:11.329 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks 2020-10-15T16:16:11.330 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-10-15T16:16:11.331 INFO:teuthology.task.internal:Checking packages... 2020-10-15T16:16:11.350 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1' 2020-10-15T16:16:11.351 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-10-15T16:16:11.351 INFO:teuthology.packaging:ref: None 2020-10-15T16:16:11.352 INFO:teuthology.packaging:tag: None 2020-10-15T16:16:11.352 INFO:teuthology.packaging:branch: wip-yuval-cls-fifo 2020-10-15T16:16:11.352 INFO:teuthology.packaging:sha1: 417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 2020-10-15T16:16:11.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuval-cls-fifo 2020-10-15T16:16:11.525 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:16:11.526 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-10-15T16:16:11.537 INFO:teuthology.task.internal:no buildpackages task found 2020-10-15T16:16:11.537 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-10-15T16:16:11.556 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-10-15T16:16:11.557 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-10-15T16:16:11.646 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:16:21.664 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:16:31.677 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:16:41.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:16:51.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:17:01.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:17:11.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:17:21.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:17:31.786 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:17:41.804 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:17:51.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:18:01.839 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:18:11.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:18:21.874 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:18:31.889 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:18:41.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:18:51.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:19:01.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:19:11.966 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:19:21.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:19:32.001 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:19:42.021 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:19:52.040 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:20:02.058 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:20:12.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:20:22.087 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:20:32.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:20:42.116 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:20:52.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:21:02.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:21:12.167 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:21:22.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:21:32.204 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:21:42.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:21:52.236 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:22:02.251 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:22:12.266 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:22:22.281 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:22:32.299 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:22:42.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:22:52.386 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:23:02.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:23:12.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:23:22.430 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:23:32.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:23:42.458 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:23:52.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-15T16:24:02.493 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:24:12.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-15T16:24:22.522 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuvalif@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098'} 2020-10-15T16:24:22.541 DEBUG:teuthology.lock.ops:locked ubuntu@smithi183.front.sepia.ceph.com, ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:24:22.594 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc77be71b70>, signals=[15]) 2020-10-15T16:24:22.793 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of ubuntu 18.04 2020-10-15T16:24:22.796 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of ubuntu 18.04 2020-10-15T16:24:23.084 INFO:teuthology.orchestra.console:Power off smithi038 2020-10-15T16:24:23.085 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-10-15T16:24:23.104 INFO:teuthology.orchestra.console:Power off smithi183 2020-10-15T16:24:23.104 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-10-15T16:24:23.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:23.127 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:27.145 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:27.156 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:31.258 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:31.268 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:31.469 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2020-10-15T16:24:31.478 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2020-10-15T16:24:31.571 INFO:teuthology.orchestra.console:Power on smithi038 2020-10-15T16:24:31.571 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-10-15T16:24:31.582 INFO:teuthology.orchestra.console:Power on smithi183 2020-10-15T16:24:31.583 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-10-15T16:24:31.593 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:31.605 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:35.616 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:35.627 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:39.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:39.743 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-15T16:24:39.945 INFO:teuthology.orchestra.console:Power on for smithi038 completed 2020-10-15T16:24:39.953 INFO:teuthology.orchestra.console:Power on for smithi183 completed 2020-10-15T16:24:40.047 INFO:teuthology.provision.fog.smithi038:Waiting for deploy to finish 2020-10-15T16:24:40.055 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2020-10-15T16:27:26.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:27:26.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:27:26.347 ERROR:teuthology.orchestra.connection:Error connecting to smithi183.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 740, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1580, in auth_publickey return self.auth_handler.wait_for_response(my_event) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response raise e paramiko.ssh_exception.AuthenticationException: Authentication failed. 2020-10-15T16:27:50.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:27:59.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:28:08.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:28:17.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:28:33.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:28:49.363 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-15T16:28:55.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:28:55.376 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-15T16:28:57.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:01.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:01.389 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-15T16:29:06.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:07.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:07.401 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-15T16:29:13.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:13.490 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-10-15T16:29:13.493 INFO:teuthology.orchestra.run.smithi038:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-10-15T16:29:13.883 INFO:teuthology.provision.fog.smithi038:Node is ready 2020-10-15T16:29:13.884 INFO:teuthology.orchestra.run.smithi038:> hostname 2020-10-15T16:29:13.929 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038.front.sepia.ceph.com 2020-10-15T16:29:13.930 INFO:teuthology.orchestra.run.smithi038:> grep smithi038.front.sepia.ceph.com /etc/hosts 2020-10-15T16:29:13.978 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38 smithi038.front.sepia.ceph.com smithi038 2020-10-15T16:29:13.978 INFO:teuthology.orchestra.run.smithi038:> sudo hostname smithi038 2020-10-15T16:29:14.117 INFO:teuthology.orchestra.run.smithi038:> sudo sed -i -e 's/smithi038.front.sepia.ceph.com/smithi038/g' /etc/hosts 2020-10-15T16:29:14.130 INFO:teuthology.orchestra.run.smithi038:> sudo sed -i -e 's/172.21.15.38/172.21.15.38/g' /etc/hosts 2020-10-15T16:29:14.187 INFO:teuthology.orchestra.run.smithi038:> sudo sed -i -e 's/smithi038.front.sepia.ceph.com/smithi038/g' /etc/hostname 2020-10-15T16:29:14.240 INFO:teuthology.orchestra.run.smithi038:> sudo hostname smithi038 2020-10-15T16:29:14.297 INFO:teuthology.provision.fog.smithi038:Deploy complete! 2020-10-15T16:29:14.297 DEBUG:teuthology.parallel:result is None 2020-10-15T16:29:16.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:25.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:25.101 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-15T16:29:31.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:31.217 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-10-15T16:29:31.217 INFO:teuthology.orchestra.run.smithi183:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-10-15T16:29:31.667 INFO:teuthology.provision.fog.smithi183:Node is ready 2020-10-15T16:29:31.667 INFO:teuthology.orchestra.run.smithi183:> hostname 2020-10-15T16:29:31.712 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com 2020-10-15T16:29:31.713 INFO:teuthology.orchestra.run.smithi183:> grep smithi183.front.sepia.ceph.com /etc/hosts 2020-10-15T16:29:31.761 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183 2020-10-15T16:29:31.762 INFO:teuthology.orchestra.run.smithi183:> sudo hostname smithi183 2020-10-15T16:29:31.822 INFO:teuthology.orchestra.run.smithi183:> sudo sed -i -e 's/smithi183.front.sepia.ceph.com/smithi183/g' /etc/hosts 2020-10-15T16:29:31.879 INFO:teuthology.orchestra.run.smithi183:> sudo sed -i -e 's/172.21.15.183/172.21.15.183/g' /etc/hosts 2020-10-15T16:29:31.934 INFO:teuthology.orchestra.run.smithi183:> sudo sed -i -e 's/smithi183.front.sepia.ceph.com/smithi183/g' /etc/hostname 2020-10-15T16:29:31.988 INFO:teuthology.orchestra.run.smithi183:> sudo hostname smithi183 2020-10-15T16:29:32.045 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2020-10-15T16:29:32.045 DEBUG:teuthology.parallel:result is None 2020-10-15T16:29:32.174 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com 2020-10-15T16:29:32.175 INFO:teuthology.lock.ops:New key found. Updating... 2020-10-15T16:29:32.191 INFO:teuthology.lock.ops:Checking smithi038.front.sepia.ceph.com 2020-10-15T16:29:32.192 INFO:teuthology.lock.ops:New key found. Updating... 2020-10-15T16:29:32.209 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server 2020-10-15T16:29:32.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:29:32.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:32.284 INFO:teuthology.orchestra.run.smithi183:> true 2020-10-15T16:29:32.471 INFO:teuthology.orchestra.run.smithi183:> uname -m 2020-10-15T16:29:32.516 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2020-10-15T16:29:32.517 INFO:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2020-10-15T16:29:32.565 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu" 2020-10-15T16:29:32.565 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-15T16:29:32.565 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu 2020-10-15T16:29:32.565 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="18.04" 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-15T16:29:32.566 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=bionic 2020-10-15T16:29:32.567 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=bionic 2020-10-15T16:29:32.568 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2020-10-15T16:29:32.583 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: set os type and version on server 2020-10-15T16:29:32.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:29:32.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:32.654 INFO:teuthology.orchestra.run.smithi038:> true 2020-10-15T16:29:32.847 INFO:teuthology.orchestra.run.smithi038:> uname -m 2020-10-15T16:29:32.895 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2020-10-15T16:29:32.895 INFO:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2020-10-15T16:29:32.943 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="Ubuntu" 2020-10-15T16:29:32.943 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-15T16:29:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:ID=ubuntu 2020-10-15T16:29:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE=debian 2020-10-15T16:29:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-15T16:29:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="18.04" 2020-10-15T16:29:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-15T16:29:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-15T16:29:32.945 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-15T16:29:32.945 INFO:teuthology.orchestra.run.smithi038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-15T16:29:32.945 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_CODENAME=bionic 2020-10-15T16:29:32.945 INFO:teuthology.orchestra.run.smithi038.stdout:UBUNTU_CODENAME=bionic 2020-10-15T16:29:32.945 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2020-10-15T16:29:32.961 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-10-15T16:29:32.993 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1tZHJ9OhSMJ524FtRmYYH9mKdurGuoC1xzwIc+OnxZ3GQCn/VLl87OsLJ5F7GkCFOSabal/LT3fLJ+GztdJk8iEFZjn/LVrGWshqgyATInaHRf4KJdVRNBS2Yhvg9916OMVHQYHUbUPFwKjfsqCVQmEBc0dAYvsJH8IqpR7t0ByfgsY0A5MyGUdo/qHiq7j3yYU9yIr4m+mPiK6x2d3RGgkR6OO9zU86kRU6R8zBN7Yne6UNBnR3WWsEivDppY1cAb9Ty1XhLMRpiEd/cGxVRidv+RY8cXCFzf9hg6YPnkcSnszdXVEYXod/2Oau63yeM5L70mTdah/Gmr4E4qRP1 smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyyo5jDAgNCY2MAVJg7aLSIIrwNjaSiAqJyM9dxk4rMvo/VKE1meyht5fMhq+77fogrBP3Y8nfI9WRD1WvifWzXuLSftj7hTCcPEHJDRTUwBNeyg2FHMne2CrEAERvBNvkJkIRp79EeozjMGpNZ4Dxz8cG0kGiaUcCaWSprB3pLuCtg0oHKJ9M/C98g4fI4uXWmyfQNx5XgJJSJ5EOJcNw5jWS1xoq61U8APotg6FyxMSTh/YWpDb7wf1GHgxNFvQS+MhgWvZxR25aJ4wYev6HPy6neeVTlnbJH1SqCn7ChVswNcyVDMR6a5OTd4QMDLs+TIk7cnNN0TLF0mvlUCr3 2020-10-15T16:29:32.993 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-10-15T16:29:33.044 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-10-15T16:29:33.060 INFO:teuthology.task.internal:Saving configuration 2020-10-15T16:29:33.076 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-10-15T16:29:33.088 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-10-15T16:29:33.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-10-15 16:24:22.528178', 'description': '/home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBNVCWO61yKDaYrnxRA+t+q+fL/6jTsBydyIdBzloJfn', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi183.front.sepia.ceph.com'} 2020-10-15T16:29:33.121 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-10-15 16:24:22.528196', 'description': '/home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDo0g3Ij+sRFmQAeLxrfMyAGZMoLrPPOUkOQfXX/MrCi', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi038.front.sepia.ceph.com'} 2020-10-15T16:29:33.121 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-10-15T16:29:33.132 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-10-15T16:29:33.132 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-10-15T16:29:33.132 INFO:teuthology.run_tasks:Running task console_log... 2020-10-15T16:29:33.198 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc77fb8c9d8>, signals=[15]) 2020-10-15T16:29:33.198 INFO:teuthology.run_tasks:Running task internal.connect... 2020-10-15T16:29:33.208 INFO:teuthology.task.internal:Opening connections... 2020-10-15T16:29:33.209 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:29:33.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:33.284 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:29:33.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:29:33.362 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-10-15T16:29:33.372 INFO:teuthology.orchestra.run.smithi183:> uname -m 2020-10-15T16:29:33.480 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2020-10-15T16:29:33.481 INFO:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2020-10-15T16:29:33.533 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu" 2020-10-15T16:29:33.533 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-15T16:29:33.533 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu 2020-10-15T16:29:33.533 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian 2020-10-15T16:29:33.533 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-15T16:29:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="18.04" 2020-10-15T16:29:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-15T16:29:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-15T16:29:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-15T16:29:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-15T16:29:33.534 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=bionic 2020-10-15T16:29:33.535 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=bionic 2020-10-15T16:29:33.535 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2020-10-15T16:29:33.551 INFO:teuthology.orchestra.run.smithi038:> uname -m 2020-10-15T16:29:33.572 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2020-10-15T16:29:33.572 INFO:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2020-10-15T16:29:33.619 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="Ubuntu" 2020-10-15T16:29:33.619 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-15T16:29:33.619 INFO:teuthology.orchestra.run.smithi038.stdout:ID=ubuntu 2020-10-15T16:29:33.619 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE=debian 2020-10-15T16:29:33.620 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-15T16:29:33.620 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="18.04" 2020-10-15T16:29:33.620 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-15T16:29:33.620 INFO:teuthology.orchestra.run.smithi038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-15T16:29:33.620 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-15T16:29:33.621 INFO:teuthology.orchestra.run.smithi038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-15T16:29:33.621 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_CODENAME=bionic 2020-10-15T16:29:33.621 INFO:teuthology.orchestra.run.smithi038.stdout:UBUNTU_CODENAME=bionic 2020-10-15T16:29:33.621 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2020-10-15T16:29:33.635 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-10-15T16:29:33.672 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-10-15T16:29:33.682 INFO:teuthology.task.internal:Checking for old test directory... 2020-10-15T16:29:33.683 INFO:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2020-10-15T16:29:33.685 INFO:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2020-10-15T16:29:33.690 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-10-15T16:29:33.701 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-10-15T16:29:33.701 INFO:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2020-10-15T16:29:33.731 INFO:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2020-10-15T16:29:33.739 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-10-15T16:29:33.781 INFO:teuthology.run_tasks:Running task kernel... 2020-10-15T16:29:33.798 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-10-15T16:29:33.798 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-10-15T16:29:33.799 INFO:teuthology.orchestra.run.smithi183:> uname -r 2020-10-15T16:29:33.804 INFO:teuthology.orchestra.run.smithi183.stdout:4.15.0-118-generic 2020-10-15T16:29:33.805 INFO:teuthology.task.kernel:Running kernel on smithi183: 4.15.0-118-generic 2020-10-15T16:29:33.805 INFO:teuthology.orchestra.run.smithi183:> sudo apt-get clean 2020-10-15T16:29:33.912 INFO:teuthology.orchestra.run.smithi183:> sudo apt-get update 2020-10-15T16:29:34.158 INFO:teuthology.orchestra.run.smithi183.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-10-15T16:29:34.173 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-15T16:29:34.240 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-10-15T16:29:34.516 INFO:teuthology.orchestra.run.smithi183.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-10-15T16:29:34.557 INFO:teuthology.orchestra.run.smithi183.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [838 kB] 2020-10-15T16:29:34.679 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,133 kB] 2020-10-15T16:29:34.932 INFO:teuthology.orchestra.run.smithi183.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,387 kB] 2020-10-15T16:29:35.028 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,713 kB] 2020-10-15T16:29:35.044 INFO:teuthology.orchestra.run.smithi183.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [274 kB] 2020-10-15T16:29:35.060 INFO:teuthology.orchestra.run.smithi183.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [160 kB] 2020-10-15T16:29:35.069 INFO:teuthology.orchestra.run.smithi183.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [21.5 kB] 2020-10-15T16:29:35.070 INFO:teuthology.orchestra.run.smithi183.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [962 kB] 2020-10-15T16:29:35.109 INFO:teuthology.orchestra.run.smithi183.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,075 kB] 2020-10-15T16:29:35.129 INFO:teuthology.orchestra.run.smithi183.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [364 kB] 2020-10-15T16:29:35.141 INFO:teuthology.orchestra.run.smithi183.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [240 kB] 2020-10-15T16:29:35.146 INFO:teuthology.orchestra.run.smithi183.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.2 kB] 2020-10-15T16:29:35.150 INFO:teuthology.orchestra.run.smithi183.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [178 kB] 2020-10-15T16:29:35.152 INFO:teuthology.orchestra.run.smithi183.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [12.6 kB] 2020-10-15T16:29:35.152 INFO:teuthology.orchestra.run.smithi183.stdout:Get:19 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,936 B] 2020-10-15T16:29:35.155 INFO:teuthology.orchestra.run.smithi183.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [24.0 kB] 2020-10-15T16:29:35.156 INFO:teuthology.orchestra.run.smithi183.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,678 kB] 2020-10-15T16:29:35.202 INFO:teuthology.orchestra.run.smithi183.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,543 kB] 2020-10-15T16:29:35.241 INFO:teuthology.orchestra.run.smithi183.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [353 kB] 2020-10-15T16:29:35.265 INFO:teuthology.orchestra.run.smithi183.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [12.9 kB] 2020-10-15T16:29:35.265 INFO:teuthology.orchestra.run.smithi183.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.6 kB] 2020-10-15T16:29:35.266 INFO:teuthology.orchestra.run.smithi183.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,952 B] 2020-10-15T16:29:36.529 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 12.3 MB in 2s (5,142 kB/s) 2020-10-15T16:29:37.909 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:29:37.955 INFO:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-10-15T16:29:38.035 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:29:38.291 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2020-10-15T16:29:38.292 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2020-10-15T16:29:38.370 INFO:teuthology.orchestra.run.smithi183.stderr:E: Unable to locate package linux-image-current-generic 2020-10-15T16:29:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-10-15T16:29:38.372 INFO:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-10-15T16:29:38.448 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:29:38.676 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2020-10-15T16:29:38.677 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2020-10-15T16:29:38.849 INFO:teuthology.orchestra.run.smithi183.stdout:The following additional packages will be installed: 2020-10-15T16:29:38.849 INFO:teuthology.orchestra.run.smithi183.stdout: linux-generic linux-headers-4.15.0-121 linux-headers-4.15.0-121-generic 2020-10-15T16:29:38.849 INFO:teuthology.orchestra.run.smithi183.stdout: linux-headers-generic linux-image-4.15.0-121-generic 2020-10-15T16:29:38.849 INFO:teuthology.orchestra.run.smithi183.stdout: linux-modules-4.15.0-121-generic linux-modules-extra-4.15.0-121-generic 2020-10-15T16:29:38.851 INFO:teuthology.orchestra.run.smithi183.stdout:Suggested packages: 2020-10-15T16:29:38.851 INFO:teuthology.orchestra.run.smithi183.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-10-15T16:29:38.851 INFO:teuthology.orchestra.run.smithi183.stdout:Recommended packages: 2020-10-15T16:29:38.851 INFO:teuthology.orchestra.run.smithi183.stdout: thermald 2020-10-15T16:29:38.883 INFO:teuthology.orchestra.run.smithi183.stdout:The following NEW packages will be installed: 2020-10-15T16:29:38.883 INFO:teuthology.orchestra.run.smithi183.stdout: linux-headers-4.15.0-121 linux-headers-4.15.0-121-generic 2020-10-15T16:29:38.883 INFO:teuthology.orchestra.run.smithi183.stdout: linux-image-4.15.0-121-generic linux-modules-4.15.0-121-generic 2020-10-15T16:29:38.883 INFO:teuthology.orchestra.run.smithi183.stdout: linux-modules-extra-4.15.0-121-generic 2020-10-15T16:29:38.885 INFO:teuthology.orchestra.run.smithi183.stdout:The following packages will be upgraded: 2020-10-15T16:29:38.885 INFO:teuthology.orchestra.run.smithi183.stdout: linux-generic linux-headers-generic linux-image-generic 2020-10-15T16:29:39.063 INFO:teuthology.orchestra.run.smithi183.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2020-10-15T16:29:39.063 INFO:teuthology.orchestra.run.smithi183.stdout:Need to get 66.0 MB of archives. 2020-10-15T16:29:39.063 INFO:teuthology.orchestra.run.smithi183.stdout:After this operation, 333 MB of additional disk space will be used. 2020-10-15T16:29:39.063 INFO:teuthology.orchestra.run.smithi183.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-121-generic amd64 4.15.0-121.123 [13.1 MB] 2020-10-15T16:29:40.060 INFO:teuthology.orchestra.run.smithi183.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-121-generic amd64 4.15.0-121.123 [8,013 kB] 2020-10-15T16:29:40.299 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-121-generic amd64 4.15.0-121.123 [32.8 MB] 2020-10-15T16:29:41.269 INFO:teuthology.orchestra.run.smithi183.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.121.108 [1,872 B] 2020-10-15T16:29:41.269 INFO:teuthology.orchestra.run.smithi183.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.121.108 [2,648 B] 2020-10-15T16:29:41.270 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-121 all 4.15.0-121.123 [10.9 MB] 2020-10-15T16:29:41.621 INFO:teuthology.orchestra.run.smithi183.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-121-generic amd64 4.15.0-121.123 [1,105 kB] 2020-10-15T16:29:41.647 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.121.108 [2,544 B] 2020-10-15T16:29:42.127 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 66.0 MB in 3s (24.0 MB/s) 2020-10-15T16:29:42.286 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-modules-4.15.0-121-generic. 2020-10-15T16:29:42.322 INFO:teuthology.orchestra.run.smithi183.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102061 files and directories currently installed.) 2020-10-15T16:29:42.325 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../0-linux-modules-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:29:42.370 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-modules-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:44.037 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-image-4.15.0-121-generic. 2020-10-15T16:29:44.050 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../1-linux-image-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:29:44.134 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-image-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:44.657 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-121-generic. 2020-10-15T16:29:44.669 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:29:44.698 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-modules-extra-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:48.466 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../3-linux-generic_4.15.0.121.108_amd64.deb ... 2020-10-15T16:29:48.591 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-generic (4.15.0.121.108) over (4.15.0.118.105) ... 2020-10-15T16:29:48.940 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.121.108_amd64.deb ... 2020-10-15T16:29:49.037 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-image-generic (4.15.0.121.108) over (4.15.0.118.105) ... 2020-10-15T16:29:49.406 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-headers-4.15.0-121. 2020-10-15T16:29:49.420 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../5-linux-headers-4.15.0-121_4.15.0-121.123_all.deb ... 2020-10-15T16:29:49.464 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-headers-4.15.0-121 (4.15.0-121.123) ... 2020-10-15T16:29:54.101 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package linux-headers-4.15.0-121-generic. 2020-10-15T16:29:54.123 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../6-linux-headers-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:29:54.168 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-headers-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:56.577 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.121.108_amd64.deb ... 2020-10-15T16:29:56.700 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking linux-headers-generic (4.15.0.121.108) over (4.15.0.118.105) ... 2020-10-15T16:29:57.051 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-modules-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:57.597 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-headers-4.15.0-121 (4.15.0-121.123) ... 2020-10-15T16:29:57.723 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-headers-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:57.849 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-image-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:58.361 INFO:teuthology.orchestra.run.smithi183.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-121-generic 2020-10-15T16:29:58.362 INFO:teuthology.orchestra.run.smithi183.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-121-generic 2020-10-15T16:29:59.142 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-headers-generic (4.15.0.121.108) ... 2020-10-15T16:29:59.285 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-modules-extra-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:29:59.786 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-image-generic (4.15.0.121.108) ... 2020-10-15T16:29:59.912 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up linux-generic (4.15.0.121.108) ... 2020-10-15T16:30:00.047 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for linux-image-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:00.253 INFO:teuthology.orchestra.run.smithi183.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-10-15T16:30:00.253 INFO:teuthology.orchestra.run.smithi183.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-121-generic 2020-10-15T16:30:15.885 INFO:teuthology.orchestra.run.smithi183.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-10-15T16:30:15.885 INFO:teuthology.orchestra.run.smithi183.stdout:Sourcing file `/etc/default/grub' 2020-10-15T16:30:15.925 INFO:teuthology.orchestra.run.smithi183.stdout:Generating grub configuration file ... 2020-10-15T16:30:16.357 INFO:teuthology.orchestra.run.smithi183.stdout:Found linux image: /boot/vmlinuz-4.15.0-121-generic 2020-10-15T16:30:16.392 INFO:teuthology.orchestra.run.smithi183.stdout:Found initrd image: /boot/initrd.img-4.15.0-121-generic 2020-10-15T16:30:16.804 INFO:teuthology.orchestra.run.smithi183.stdout:Found linux image: /boot/vmlinuz-4.15.0-118-generic 2020-10-15T16:30:16.831 INFO:teuthology.orchestra.run.smithi183.stdout:Found initrd image: /boot/initrd.img-4.15.0-118-generic 2020-10-15T16:30:18.585 INFO:teuthology.orchestra.run.smithi183.stdout:done 2020-10-15T16:30:18.995 INFO:teuthology.orchestra.run.smithi183:> dpkg -s linux-image-generic 2020-10-15T16:30:19.014 INFO:teuthology.orchestra.run.smithi183.stdout:Package: linux-image-generic 2020-10-15T16:30:19.014 INFO:teuthology.orchestra.run.smithi183.stdout:Status: install ok installed 2020-10-15T16:30:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:Priority: optional 2020-10-15T16:30:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:Section: kernel 2020-10-15T16:30:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:Installed-Size: 16 2020-10-15T16:30:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:Maintainer: Ubuntu Kernel Team 2020-10-15T16:30:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: amd64 2020-10-15T16:30:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:Source: linux-meta 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout:Version: 4.15.0.121.108 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout:Depends: linux-image-4.15.0-121-generic, linux-modules-extra-4.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout:Recommends: thermald 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout:Description: Generic Linux kernel image 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: This package will always depend on the latest generic kernel image 2020-10-15T16:30:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: available. 2020-10-15T16:30:19.017 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-118-generic Expected: 4.15.0-121-generic 2020-10-15T16:30:19.017 INFO:teuthology.orchestra.run.smithi038:> uname -r 2020-10-15T16:30:19.023 INFO:teuthology.orchestra.run.smithi038.stdout:4.15.0-118-generic 2020-10-15T16:30:19.024 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.15.0-118-generic 2020-10-15T16:30:19.024 INFO:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2020-10-15T16:30:19.146 INFO:teuthology.orchestra.run.smithi038:> sudo apt-get update 2020-10-15T16:30:19.412 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-15T16:30:19.412 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-10-15T16:30:19.476 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-10-15T16:30:19.823 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-10-15T16:30:19.832 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,387 kB] 2020-10-15T16:30:19.968 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,713 kB] 2020-10-15T16:30:20.268 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [838 kB] 2020-10-15T16:30:20.303 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [274 kB] 2020-10-15T16:30:20.315 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [160 kB] 2020-10-15T16:30:20.321 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [21.5 kB] 2020-10-15T16:30:20.322 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,075 kB] 2020-10-15T16:30:20.357 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,133 kB] 2020-10-15T16:30:20.376 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [364 kB] 2020-10-15T16:30:20.381 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.2 kB] 2020-10-15T16:30:20.382 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [178 kB] 2020-10-15T16:30:20.383 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [24.0 kB] 2020-10-15T16:30:20.383 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,543 kB] 2020-10-15T16:30:20.386 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [962 kB] 2020-10-15T16:30:20.415 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [240 kB] 2020-10-15T16:30:20.419 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [12.6 kB] 2020-10-15T16:30:20.419 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,936 B] 2020-10-15T16:30:20.460 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,678 kB] 2020-10-15T16:30:20.528 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [353 kB] 2020-10-15T16:30:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.6 kB] 2020-10-15T16:30:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [12.9 kB] 2020-10-15T16:30:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,952 B] 2020-10-15T16:30:21.918 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 12.3 MB in 3s (4,814 kB/s) 2020-10-15T16:30:23.391 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:30:23.441 INFO:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-10-15T16:30:23.523 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:30:23.762 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2020-10-15T16:30:23.763 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2020-10-15T16:30:23.853 INFO:teuthology.orchestra.run.smithi038.stderr:E: Unable to locate package linux-image-current-generic 2020-10-15T16:30:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-10-15T16:30:23.855 INFO:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-10-15T16:30:23.937 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:30:24.175 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2020-10-15T16:30:24.176 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2020-10-15T16:30:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2020-10-15T16:30:24.365 INFO:teuthology.orchestra.run.smithi038.stdout: linux-generic linux-headers-4.15.0-121 linux-headers-4.15.0-121-generic 2020-10-15T16:30:24.365 INFO:teuthology.orchestra.run.smithi038.stdout: linux-headers-generic linux-image-4.15.0-121-generic 2020-10-15T16:30:24.366 INFO:teuthology.orchestra.run.smithi038.stdout: linux-modules-4.15.0-121-generic linux-modules-extra-4.15.0-121-generic 2020-10-15T16:30:24.372 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2020-10-15T16:30:24.372 INFO:teuthology.orchestra.run.smithi038.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-10-15T16:30:24.372 INFO:teuthology.orchestra.run.smithi038.stdout:Recommended packages: 2020-10-15T16:30:24.372 INFO:teuthology.orchestra.run.smithi038.stdout: thermald 2020-10-15T16:30:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2020-10-15T16:30:24.402 INFO:teuthology.orchestra.run.smithi038.stdout: linux-headers-4.15.0-121 linux-headers-4.15.0-121-generic 2020-10-15T16:30:24.402 INFO:teuthology.orchestra.run.smithi038.stdout: linux-image-4.15.0-121-generic linux-modules-4.15.0-121-generic 2020-10-15T16:30:24.402 INFO:teuthology.orchestra.run.smithi038.stdout: linux-modules-extra-4.15.0-121-generic 2020-10-15T16:30:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages will be upgraded: 2020-10-15T16:30:24.404 INFO:teuthology.orchestra.run.smithi038.stdout: linux-generic linux-headers-generic linux-image-generic 2020-10-15T16:30:24.583 INFO:teuthology.orchestra.run.smithi038.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2020-10-15T16:30:24.583 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 66.0 MB of archives. 2020-10-15T16:30:24.584 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 333 MB of additional disk space will be used. 2020-10-15T16:30:24.584 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-121-generic amd64 4.15.0-121.123 [13.1 MB] 2020-10-15T16:30:25.586 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-121-generic amd64 4.15.0-121.123 [8,013 kB] 2020-10-15T16:30:25.827 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-121-generic amd64 4.15.0-121.123 [32.8 MB] 2020-10-15T16:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.121.108 [1,872 B] 2020-10-15T16:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.121.108 [2,648 B] 2020-10-15T16:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-121 all 4.15.0-121.123 [10.9 MB] 2020-10-15T16:30:27.204 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-121-generic amd64 4.15.0-121.123 [1,105 kB] 2020-10-15T16:30:27.231 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.121.108 [2,544 B] 2020-10-15T16:30:27.732 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 66.0 MB in 3s (23.5 MB/s) 2020-10-15T16:30:27.897 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-modules-4.15.0-121-generic. 2020-10-15T16:30:27.935 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102061 files and directories currently installed.) 2020-10-15T16:30:27.937 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../0-linux-modules-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:30:27.981 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-modules-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:29.625 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-image-4.15.0-121-generic. 2020-10-15T16:30:29.638 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../1-linux-image-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:30:29.730 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-image-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:30.245 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-121-generic. 2020-10-15T16:30:30.256 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:30:30.295 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-modules-extra-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:33.957 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../3-linux-generic_4.15.0.121.108_amd64.deb ... 2020-10-15T16:30:34.075 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-generic (4.15.0.121.108) over (4.15.0.118.105) ... 2020-10-15T16:30:34.425 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.121.108_amd64.deb ... 2020-10-15T16:30:34.546 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-image-generic (4.15.0.121.108) over (4.15.0.118.105) ... 2020-10-15T16:30:34.865 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-headers-4.15.0-121. 2020-10-15T16:30:34.878 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../5-linux-headers-4.15.0-121_4.15.0-121.123_all.deb ... 2020-10-15T16:30:34.915 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-headers-4.15.0-121 (4.15.0-121.123) ... 2020-10-15T16:30:39.634 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-headers-4.15.0-121-generic. 2020-10-15T16:30:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../6-linux-headers-4.15.0-121-generic_4.15.0-121.123_amd64.deb ... 2020-10-15T16:30:39.684 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-headers-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:42.132 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.121.108_amd64.deb ... 2020-10-15T16:30:42.237 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-headers-generic (4.15.0.121.108) over (4.15.0.118.105) ... 2020-10-15T16:30:42.563 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-modules-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:43.126 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-headers-4.15.0-121 (4.15.0-121.123) ... 2020-10-15T16:30:43.235 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-headers-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:43.336 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-image-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:43.789 INFO:teuthology.orchestra.run.smithi038.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-121-generic 2020-10-15T16:30:43.789 INFO:teuthology.orchestra.run.smithi038.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-121-generic 2020-10-15T16:30:43.930 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-headers-generic (4.15.0.121.108) ... 2020-10-15T16:30:44.057 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-modules-extra-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:44.558 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-image-generic (4.15.0.121.108) ... 2020-10-15T16:30:44.684 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-generic (4.15.0.121.108) ... 2020-10-15T16:30:44.802 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for linux-image-4.15.0-121-generic (4.15.0-121.123) ... 2020-10-15T16:30:44.990 INFO:teuthology.orchestra.run.smithi038.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-10-15T16:30:44.990 INFO:teuthology.orchestra.run.smithi038.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-121-generic 2020-10-15T16:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-10-15T16:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout:Sourcing file `/etc/default/grub' 2020-10-15T16:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout:Generating grub configuration file ... 2020-10-15T16:31:00.842 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-4.15.0-121-generic 2020-10-15T16:31:00.875 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-4.15.0-121-generic 2020-10-15T16:31:01.240 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-4.15.0-118-generic 2020-10-15T16:31:01.274 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-4.15.0-118-generic 2020-10-15T16:31:02.928 INFO:teuthology.orchestra.run.smithi038.stdout:done 2020-10-15T16:31:03.344 INFO:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic 2020-10-15T16:31:03.368 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic 2020-10-15T16:31:03.368 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2020-10-15T16:31:03.368 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 16 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 4.15.0.121.108 2020-10-15T16:31:03.369 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-10-15T16:31:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-4.15.0-121-generic, linux-modules-extra-4.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2020-10-15T16:31:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2020-10-15T16:31:03.370 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2020-10-15T16:31:03.370 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2020-10-15T16:31:03.370 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2020-10-15T16:31:03.371 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-118-generic Expected: 4.15.0-121-generic 2020-10-15T16:31:03.371 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-10-15T16:31:03.371 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-10-15T16:31:03.371 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-10-15T16:31:03.372 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-10-15T16:31:03.373 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2020-10-15T16:31:03.373 INFO:teuthology.orchestra.run.smithi183:> sudo apt-get clean 2020-10-15T16:31:03.484 INFO:teuthology.orchestra.run.smithi183:> sudo apt-get update 2020-10-15T16:31:03.626 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-10-15T16:31:03.742 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-15T16:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-10-15T16:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-10-15T16:31:05.475 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:31:05.519 INFO:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-10-15T16:31:05.598 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:31:05.832 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2020-10-15T16:31:05.832 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2020-10-15T16:31:05.911 INFO:teuthology.orchestra.run.smithi183.stderr:E: Unable to locate package linux-image-current-generic 2020-10-15T16:31:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-10-15T16:31:05.913 INFO:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-10-15T16:31:05.990 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:31:06.220 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2020-10-15T16:31:06.221 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2020-10-15T16:31:06.413 INFO:teuthology.orchestra.run.smithi183.stdout:linux-image-generic is already the newest version (4.15.0.121.108). 2020-10-15T16:31:06.414 INFO:teuthology.orchestra.run.smithi183.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2020-10-15T16:31:06.415 INFO:teuthology.orchestra.run.smithi183:> dpkg -s linux-image-generic 2020-10-15T16:31:06.430 INFO:teuthology.orchestra.run.smithi183.stdout:Package: linux-image-generic 2020-10-15T16:31:06.430 INFO:teuthology.orchestra.run.smithi183.stdout:Status: install ok installed 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Priority: optional 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Section: kernel 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Installed-Size: 16 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Maintainer: Ubuntu Kernel Team 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: amd64 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Source: linux-meta 2020-10-15T16:31:06.431 INFO:teuthology.orchestra.run.smithi183.stdout:Version: 4.15.0.121.108 2020-10-15T16:31:06.432 INFO:teuthology.orchestra.run.smithi183.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-10-15T16:31:06.432 INFO:teuthology.orchestra.run.smithi183.stdout:Depends: linux-image-4.15.0-121-generic, linux-modules-extra-4.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2020-10-15T16:31:06.432 INFO:teuthology.orchestra.run.smithi183.stdout:Recommends: thermald 2020-10-15T16:31:06.432 INFO:teuthology.orchestra.run.smithi183.stdout:Description: Generic Linux kernel image 2020-10-15T16:31:06.432 INFO:teuthology.orchestra.run.smithi183.stdout: This package will always depend on the latest generic kernel image 2020-10-15T16:31:06.432 INFO:teuthology.orchestra.run.smithi183.stdout: available. 2020-10-15T16:31:06.433 INFO:teuthology.orchestra.run.smithi183:> mktemp 2020-10-15T16:31:06.478 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.ExdxuGLNNO 2020-10-15T16:31:06.479 INFO:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ExdxuGLNNO 2020-10-15T16:31:06.550 INFO:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.ExdxuGLNNO 2020-10-15T16:31:06.675 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.ExdxuGLNNO is 8KB 2020-10-15T16:31:06.686 INFO:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.ExdxuGLNNO 2020-10-15T16:31:06.693 INFO:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-10-15T16:31:06.747 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:31:06.747 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-10-15T16:31:06.748 INFO:teuthology.orchestra.run.smithi183:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-10-15T16:31:06.793 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-10-15T16:31:06.814 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:31:06.814 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:31:06.815 INFO:teuthology.orchestra.run.smithi183.stderr:93 bytes copied, 3.4655e-05 s, 2.7 MB/s 2020-10-15T16:31:06.815 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-10-15T16:31:06.825 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-121-generic 2020-10-15T16:31:06.825 INFO:teuthology.orchestra.run.smithi183:> sudo update-grub 2020-10-15T16:31:08.093 INFO:teuthology.orchestra.run.smithi183.stderr:Sourcing file `/etc/default/grub' 2020-10-15T16:31:08.125 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2020-10-15T16:31:08.415 INFO:teuthology.orchestra.run.smithi183.stderr:Found linux image: /boot/vmlinuz-4.15.0-121-generic 2020-10-15T16:31:08.427 INFO:teuthology.orchestra.run.smithi183.stderr:Found initrd image: /boot/initrd.img-4.15.0-121-generic 2020-10-15T16:31:08.761 INFO:teuthology.orchestra.run.smithi183.stderr:Found linux image: /boot/vmlinuz-4.15.0-118-generic 2020-10-15T16:31:08.773 INFO:teuthology.orchestra.run.smithi183.stderr:Found initrd image: /boot/initrd.img-4.15.0-118-generic 2020-10-15T16:31:09.982 INFO:teuthology.orchestra.run.smithi183.stderr:done 2020-10-15T16:31:09.983 INFO:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2020-10-15T16:31:09.985 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-10-15T16:31:09.986 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2020-10-15T16:31:09.986 INFO:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2020-10-15T16:31:10.048 INFO:teuthology.orchestra.run.smithi038:> sudo apt-get update 2020-10-15T16:31:10.173 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-10-15T16:31:10.289 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-15T16:31:10.369 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-10-15T16:31:10.455 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-10-15T16:31:12.084 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:31:12.133 INFO:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-10-15T16:31:12.215 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:31:12.462 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2020-10-15T16:31:12.463 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2020-10-15T16:31:12.554 INFO:teuthology.orchestra.run.smithi038.stderr:E: Unable to locate package linux-image-current-generic 2020-10-15T16:31:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-10-15T16:31:12.556 INFO:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-10-15T16:31:12.636 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:31:12.882 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2020-10-15T16:31:12.883 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2020-10-15T16:31:13.103 INFO:teuthology.orchestra.run.smithi038.stdout:linux-image-generic is already the newest version (4.15.0.121.108). 2020-10-15T16:31:13.104 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2020-10-15T16:31:13.105 INFO:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic 2020-10-15T16:31:13.128 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic 2020-10-15T16:31:13.128 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2020-10-15T16:31:13.128 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2020-10-15T16:31:13.128 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 16 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 4.15.0.121.108 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-10-15T16:31:13.129 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-4.15.0-121-generic, linux-modules-extra-4.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2020-10-15T16:31:13.130 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2020-10-15T16:31:13.130 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2020-10-15T16:31:13.130 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2020-10-15T16:31:13.130 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2020-10-15T16:31:13.130 INFO:teuthology.orchestra.run.smithi038:> mktemp 2020-10-15T16:31:13.176 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.cyk9WnEAuz 2020-10-15T16:31:13.177 INFO:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cyk9WnEAuz 2020-10-15T16:31:13.231 INFO:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.cyk9WnEAuz 2020-10-15T16:31:13.347 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.cyk9WnEAuz is 8KB 2020-10-15T16:31:13.357 INFO:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.cyk9WnEAuz 2020-10-15T16:31:13.363 INFO:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-10-15T16:31:13.420 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:31:13.420 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-10-15T16:31:13.420 INFO:teuthology.orchestra.run.smithi038:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-10-15T16:31:13.466 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-10-15T16:31:13.482 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:31:13.482 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:31:13.482 INFO:teuthology.orchestra.run.smithi038.stderr:93 bytes copied, 3.4046e-05 s, 2.7 MB/s 2020-10-15T16:31:13.483 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-10-15T16:31:13.492 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-121-generic 2020-10-15T16:31:13.492 INFO:teuthology.orchestra.run.smithi038:> sudo update-grub 2020-10-15T16:31:14.818 INFO:teuthology.orchestra.run.smithi038.stderr:Sourcing file `/etc/default/grub' 2020-10-15T16:31:14.832 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2020-10-15T16:31:15.119 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-4.15.0-121-generic 2020-10-15T16:31:15.133 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-4.15.0-121-generic 2020-10-15T16:31:15.471 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-4.15.0-118-generic 2020-10-15T16:31:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-4.15.0-118-generic 2020-10-15T16:31:16.905 INFO:teuthology.orchestra.run.smithi038.stderr:done 2020-10-15T16:31:16.907 INFO:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2020-10-15T16:31:46.909 INFO:teuthology.misc:Re-opening connections... 2020-10-15T16:31:46.910 INFO:teuthology.misc:trying to connect to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:31:46.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:31:46.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:05.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2020-10-15T16:32:05.443 INFO:teuthology.misc:trying to connect to ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:32:05.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:05.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:39.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2020-10-15T16:32:39.999 DEBUG:teuthology.misc:waited 53.088467597961426 2020-10-15T16:32:41.000 INFO:teuthology.misc:trying to connect to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:32:41.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:41.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:44.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2020-10-15T16:32:44.063 INFO:teuthology.misc:trying to connect to ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:32:44.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:44.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:44.065 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2020-10-15T16:32:44.065 DEBUG:teuthology.misc:waited 57.15498065948486 2020-10-15T16:32:45.066 INFO:teuthology.misc:trying to connect to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:32:45.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:45.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:47.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2020-10-15T16:32:47.135 INFO:teuthology.misc:trying to connect to ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:32:47.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:47.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:47.136 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2020-10-15T16:32:47.137 DEBUG:teuthology.misc:waited 60.22678089141846 2020-10-15T16:32:48.137 INFO:teuthology.misc:trying to connect to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:32:48.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:48.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:50.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2020-10-15T16:32:50.206 INFO:teuthology.misc:trying to connect to ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:32:50.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:50.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:50.302 INFO:teuthology.orchestra.run.smithi038:> true 2020-10-15T16:32:50.656 DEBUG:teuthology.misc:waited 63.746350049972534 2020-10-15T16:32:51.657 INFO:teuthology.misc:trying to connect to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:32:51.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:51.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:51.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2020-10-15T16:32:51.660 DEBUG:teuthology.misc:waited 64.75007605552673 2020-10-15T16:32:52.661 INFO:teuthology.misc:trying to connect to ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:32:52.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:32:52.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:32:52.889 INFO:teuthology.orchestra.run.smithi183:> true 2020-10-15T16:32:53.842 DEBUG:teuthology.misc:waited 66.93152189254761 2020-10-15T16:32:54.843 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-10-15T16:32:54.843 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-121-generic"... 2020-10-15T16:32:54.844 INFO:teuthology.orchestra.run.smithi183:> uname -r 2020-10-15T16:32:54.849 INFO:teuthology.orchestra.run.smithi183.stdout:4.15.0-121-generic 2020-10-15T16:32:54.850 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-121-generic vs 4.15.0-121-generic 2020-10-15T16:32:54.850 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-10-15T16:32:54.850 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-10-15T16:32:54.850 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-10-15T16:32:54.851 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-121-generic"... 2020-10-15T16:32:54.851 INFO:teuthology.orchestra.run.smithi038:> uname -r 2020-10-15T16:32:54.857 INFO:teuthology.orchestra.run.smithi038.stdout:4.15.0-121-generic 2020-10-15T16:32:54.857 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-121-generic vs 4.15.0-121-generic 2020-10-15T16:32:54.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-10-15T16:32:54.858 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-10-15T16:32:55.859 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-10-15T16:32:55.859 INFO:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-10-15T16:32:55.906 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2020-10-15T16:32:55.922 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-10-15T16:32:55.922 INFO:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-10-15T16:32:55.961 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2020-10-15T16:32:55.989 INFO:teuthology.run_tasks:Running task internal.base... 2020-10-15T16:32:55.998 INFO:teuthology.task.internal:Creating test directory... 2020-10-15T16:32:55.999 INFO:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-10-15T16:32:56.033 INFO:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-10-15T16:32:56.039 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-10-15T16:32:56.048 INFO:teuthology.run_tasks:Running task internal.archive... 2020-10-15T16:32:56.079 INFO:teuthology.task.internal:Creating archive directory... 2020-10-15T16:32:56.079 INFO:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-10-15T16:32:56.081 INFO:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-10-15T16:32:56.105 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-10-15T16:32:56.127 INFO:teuthology.task.internal:Enabling coredump saving... 2020-10-15T16:32:56.128 INFO:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-10-15T16:32:56.132 INFO:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-10-15T16:32:56.153 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-15T16:32:56.160 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-15T16:32:56.171 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-15T16:32:56.178 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-15T16:32:56.179 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-10-15T16:32:56.191 INFO:teuthology.task.internal:Configuring sudo... 2020-10-15T16:32:56.192 INFO:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-10-15T16:32:56.204 INFO:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-10-15T16:32:56.238 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-10-15T16:32:56.274 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-10-15T16:32:56.275 INFO:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-10-15T16:32:56.277 INFO:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-10-15T16:32:56.285 INFO:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-10-15T16:32:56.337 INFO:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-10-15T16:32:56.385 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:32:56.386 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-15T16:32:56.432 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-15T16:32:56.445 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:32:56.445 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:32:56.445 INFO:teuthology.orchestra.run.smithi183.stderr:152 bytes copied, 3.3581e-05 s, 4.5 MB/s 2020-10-15T16:32:56.446 INFO:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-10-15T16:32:56.452 INFO:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-10-15T16:32:56.501 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:32:56.501 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-15T16:32:56.551 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-15T16:32:56.567 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:32:56.568 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:32:56.568 INFO:teuthology.orchestra.run.smithi038.stderr:152 bytes copied, 3.2583e-05 s, 4.7 MB/s 2020-10-15T16:32:56.569 INFO:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2020-10-15T16:32:56.617 INFO:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2020-10-15T16:32:56.726 INFO:teuthology.run_tasks:Running task internal.timer... 2020-10-15T16:32:56.736 INFO:teuthology.task.internal:Starting timer... 2020-10-15T16:32:56.736 INFO:teuthology.run_tasks:Running task pcp... 2020-10-15T16:32:56.782 INFO:teuthology.run_tasks:Running task selinux... 2020-10-15T16:32:56.830 DEBUG:teuthology.task.selinux:Excluding smithi183: OS 'ubuntu' does not support SELinux 2020-10-15T16:32:56.830 DEBUG:teuthology.task.selinux:Excluding smithi038: OS 'ubuntu' does not support SELinux 2020-10-15T16:32:56.830 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-10-15T16:32:56.831 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-10-15T16:32:56.831 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-10-15T16:32:56.831 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-10-15T16:32:56.903 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-10-15T16:32:56.909 INFO:teuthology.repo_utils:Fetching master from origin 2020-10-15T16:32:56.955 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-10-15T16:32:56.968 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-10-15T16:32:56.970 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi183.front.sepia.ceph.com,smithi038.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-10-15T16:32:57.282 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-10-15T16:32:58.046 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-10-15T16:33:00.087 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-10-15T16:33:00.105 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:00 +0000 (0:00:00.064) 0:00:00.064 ****** 2020-10-15T16:33:00.118 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:00 +0000 (0:00:00.012) 0:00:00.076 ****** 2020-10-15T16:33:01.776 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi038.front.sepia.ceph.com] 2020-10-15T16:33:01.850 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:01 +0000 (0:00:01.732) 0:00:01.808 ****** 2020-10-15T16:33:01.922 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] 2020-10-15T16:33:01.991 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:01 +0000 (0:00:00.141) 0:00:01.950 ****** 2020-10-15T16:33:02.197 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-10-15T16:33:02.199 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-10-15T16:33:02.230 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:02 +0000 (0:00:00.238) 0:00:02.188 ****** 2020-10-15T16:33:02.236 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-10-15T16:33:02.272 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:02 +0000 (0:00:00.041) 0:00:02.230 ****** 2020-10-15T16:33:02.784 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-10-15T16:33:02.786 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} changed: [smithi183.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-10-15T16:33:02.821 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:02 +0000 (0:00:00.549) 0:00:02.780 ****** 2020-10-15T16:33:02.853 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:02 +0000 (0:00:00.031) 0:00:02.812 ****** 2020-10-15T16:33:03.272 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-10-15T16:33:03.274 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.085151", "end": "2020-10-15 16:33:02.947318", "rc": 0, "start": "2020-10-15 16:33:02.862167", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-10-15T16:33:03.306 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:03 +0000 (0:00:00.452) 0:00:03.264 ****** 2020-10-15T16:33:03.344 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.175179", "end": "2020-10-15 16:33:03.523334", "rc": 0, "start": "2020-10-15 16:33:03.348155", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-10-15T16:33:03.380 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:03 +0000 (0:00:00.074) 0:00:03.339 ****** 2020-10-15T16:33:03.563 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-10-15T16:33:03.565 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:33:03.599 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:03 +0000 (0:00:00.218) 0:00:03.557 ****** 2020-10-15T16:33:03.632 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:03 +0000 (0:00:00.032) 0:00:03.590 ****** 2020-10-15T16:33:04.365 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-10-15T16:33:04.367 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602779583.6159277-152844557495717/source", "state": "file", "uid": 0} changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602779583.6488452-258237598406925/source", "state": "file", "uid": 0} 2020-10-15T16:33:04.400 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:04 +0000 (0:00:00.768) 0:00:04.359 ****** 2020-10-15T16:33:04.431 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:04 +0000 (0:00:00.030) 0:00:04.389 ****** 2020-10-15T16:33:04.707 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-10-15T16:33:04.709 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-10-15T16:33:04.710 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-10-15T16:33:04.750 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-10-15T16:33:04.752 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-10-15T16:33:04.871 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-10-15T16:33:04.878 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-10-15T16:33:05.016 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:05 +0000 (0:00:00.585) 0:00:04.975 ****** 2020-10-15T16:33:05.048 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:05 +0000 (0:00:00.031) 0:00:05.006 ****** 2020-10-15T16:33:05.737 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-10-15T16:33:05.739 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] 2020-10-15T16:33:05.813 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-10-15T16:33:05.815 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:05 +0000 (0:00:00.765) 0:00:05.772 ****** 2020-10-15T16:33:06.157 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-10-15T16:33:06.159 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.160121", "end": "2020-10-15 16:33:06.136257", "rc": 0, "start": "2020-10-15 16:33:05.976136", "stderr": "", "stderr_lines": [], "stdout": "8b571645c4accd90b6fb75de5ddc6e40874f17e2", "stdout_lines": ["8b571645c4accd90b6fb75de5ddc6e40874f17e2"]} 2020-10-15T16:33:06.192 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.378) 0:00:06.150 ****** 2020-10-15T16:33:06.316 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-10-15T16:33:06.318 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013335", "end": "2020-10-15 16:33:06.494358", "failed_when_result": false, "rc": 0, "start": "2020-10-15 16:33:06.481023", "stderr": "", "stderr_lines": [], "stdout": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "stdout_lines": ["cb3b57134780b85e843ecc462024cfc477f6bcf1"]} 2020-10-15T16:33:06.350 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.158) 0:00:06.309 ****** 2020-10-15T16:33:06.378 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-10-15T16:33:06.379 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:06.411 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.060) 0:00:06.370 ****** 2020-10-15T16:33:06.438 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-10-15T16:33:06.440 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:06.472 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.060) 0:00:06.430 ****** 2020-10-15T16:33:06.499 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-10-15T16:33:06.501 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:06.533 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.061) 0:00:06.492 ****** 2020-10-15T16:33:06.560 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-10-15T16:33:06.562 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:06.600 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.066) 0:00:06.558 ****** 2020-10-15T16:33:06.627 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-10-15T16:33:06.629 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:06.661 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.060) 0:00:06.619 ****** 2020-10-15T16:33:06.784 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-10-15T16:33:06.786 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] 2020-10-15T16:33:06.790 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-10-15T16:33:06.792 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:33:06.844 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.183) 0:00:06.803 ****** 2020-10-15T16:33:06.887 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:06 +0000 (0:00:00.042) 0:00:06.845 ****** 2020-10-15T16:33:07.028 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-15T16:33:07.029 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-10-15T16:33:07.155 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-10-15T16:33:07.294 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-10-15T16:33:07.336 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-10-15T16:33:07.337 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.269498", "end": "2020-10-15 16:33:07.308202", "rc": 0, "start": "2020-10-15 16:33:07.038704", "stderr": "", "stderr_lines": [], "stdout": "8b571645c4accd90b6fb75de5ddc6e40874f17e2", "stdout_lines": ["8b571645c4accd90b6fb75de5ddc6e40874f17e2"]} 2020-10-15T16:33:07.370 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.483) 0:00:07.329 ****** 2020-10-15T16:33:07.372 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:07.431 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-15T16:33:07.433 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-10-15T16:33:07.506 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-10-15T16:33:07.507 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.015019", "end": "2020-10-15 16:33:07.189231", "failed_when_result": false, "rc": 0, "start": "2020-10-15 16:33:07.174212", "stderr": "", "stderr_lines": [], "stdout": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "stdout_lines": ["cb3b57134780b85e843ecc462024cfc477f6bcf1"]} 2020-10-15T16:33:07.539 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.168) 0:00:07.497 ****** 2020-10-15T16:33:07.566 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-10-15T16:33:07.567 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:07.602 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.063) 0:00:07.561 ****** 2020-10-15T16:33:07.609 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-15T16:33:07.611 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-10-15T16:33:07.631 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-10-15T16:33:07.633 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:07.668 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.066) 0:00:07.627 ****** 2020-10-15T16:33:07.706 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-10-15T16:33:07.708 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:07.742 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.073) 0:00:07.701 ****** 2020-10-15T16:33:07.749 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-15T16:33:07.750 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-10-15T16:33:07.772 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-10-15T16:33:07.773 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:07.805 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.063) 0:00:07.764 ****** 2020-10-15T16:33:07.832 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-10-15T16:33:07.834 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:07.867 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:07 +0000 (0:00:00.061) 0:00:07.825 ****** 2020-10-15T16:33:07.874 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-15T16:33:07.875 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-10-15T16:33:08.001 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-10-15T16:33:08.003 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-10-15T16:33:08.036 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:08 +0000 (0:00:00.168) 0:00:07.994 ****** 2020-10-15T16:33:08.147 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-10-15T16:33:08.211 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-10-15T16:33:08.285 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-10-15T16:33:08.350 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-10-15T16:33:08.486 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-10-15T16:33:08.612 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-10-15T16:33:08.747 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-10-15T16:33:08.748 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-10-15T16:33:08.874 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-10-15T16:33:08.885 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-10-15T16:33:09.002 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-10-15T16:33:09.035 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-10-15T16:33:09.131 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-10-15T16:33:09.153 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-10-15T16:33:09.265 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-10-15T16:33:09.287 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-10-15T16:33:09.392 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-10-15T16:33:09.435 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-10-15T16:33:09.520 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-10-15T16:33:09.573 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-10-15T16:33:09.648 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-10-15T16:33:09.702 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-10-15T16:33:09.779 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-10-15T16:33:09.832 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-10-15T16:33:09.917 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-10-15T16:33:09.961 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-10-15T16:33:10.046 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-10-15T16:33:10.100 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-10-15T16:33:10.175 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-10-15T16:33:10.251 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-10-15T16:33:10.316 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-10-15T16:33:10.380 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-10-15T16:33:10.454 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-10-15T16:33:10.518 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-10-15T16:33:10.592 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-10-15T16:33:10.648 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-10-15T16:33:10.721 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-10-15T16:33:10.774 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-10-15T16:33:10.848 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-10-15T16:33:10.891 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-10-15T16:33:10.974 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-10-15T16:33:11.038 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-10-15T16:33:11.102 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-10-15T16:33:11.167 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-10-15T16:33:11.236 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-10-15T16:33:11.310 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-10-15T16:33:11.374 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-10-15T16:33:11.448 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-10-15T16:33:11.501 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-10-15T16:33:11.586 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-10-15T16:33:11.630 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-10-15T16:33:11.715 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-10-15T16:33:11.759 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-10-15T16:33:11.853 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-10-15T16:33:11.885 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-10-15T16:33:12.001 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-10-15T16:33:12.013 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-10-15T16:33:12.140 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-10-15T16:33:12.141 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-10-15T16:33:12.268 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-10-15T16:33:12.280 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-10-15T16:33:12.396 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-10-15T16:33:12.397 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-10-15T16:33:12.524 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-10-15T16:33:12.527 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-10-15T16:33:12.652 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-15T16:33:12.653 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-10-15T16:33:12.778 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-10-15T16:33:12.800 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-10-15T16:33:12.906 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-10-15T16:33:12.938 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-10-15T16:33:13.033 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-10-15T16:33:13.076 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-10-15T16:33:13.170 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-10-15T16:33:13.202 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-10-15T16:33:13.296 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-15T16:33:13.328 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-10-15T16:33:13.422 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-10-15T16:33:13.454 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-10-15T16:33:13.548 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-10-15T16:33:13.592 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-10-15T16:33:13.677 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-10-15T16:33:13.720 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-15T16:33:13.804 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-10-15T16:33:13.867 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-10-15T16:33:13.942 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-10-15T16:33:13.944 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:13.997 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-10-15T16:33:14.060 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-10-15T16:33:14.137 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-10-15T16:33:14.190 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-10-15T16:33:14.254 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-10-15T16:33:14.329 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-15T16:33:14.404 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-10-15T16:33:14.469 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-10-15T16:33:14.543 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-10-15T16:33:14.545 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:14.588 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-10-15T16:33:14.674 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-10-15T16:33:14.727 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-10-15T16:33:14.811 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-10-15T16:33:14.853 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-10-15T16:33:14.947 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-10-15T16:33:14.980 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-10-15T16:33:15.074 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-10-15T16:33:15.130 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-10-15T16:33:15.222 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-10-15T16:33:15.233 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-10-15T16:33:15.339 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-10-15T16:33:15.361 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-10-15T16:33:15.487 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-10-15T16:33:15.500 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-10-15T16:33:15.616 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-10-15T16:33:15.628 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-10-15T16:33:15.754 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-10-15T16:33:15.765 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-10-15T16:33:15.767 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:15.880 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-10-15T16:33:15.891 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-10-15T16:33:16.007 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-10-15T16:33:16.029 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-15T16:33:16.148 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-10-15T16:33:16.170 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-10-15T16:33:16.274 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-10-15T16:33:16.296 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-10-15T16:33:16.402 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-10-15T16:33:16.444 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-10-15T16:33:16.528 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-15T16:33:16.571 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-10-15T16:33:16.655 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-10-15T16:33:16.697 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-10-15T16:33:16.781 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-10-15T16:33:16.823 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-10-15T16:33:16.907 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-10-15T16:33:16.970 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-10-15T16:33:17.034 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-10-15T16:33:17.108 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-10-15T16:33:17.160 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-10-15T16:33:17.244 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-10-15T16:33:17.297 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-10-15T16:33:17.392 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-10-15T16:33:17.425 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-10-15T16:33:17.520 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-10-15T16:33:17.562 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-10-15T16:33:17.657 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-10-15T16:33:17.700 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-10-15T16:33:17.796 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-10-15T16:33:17.818 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-10-15T16:33:17.955 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-10-15T16:33:17.956 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-10-15T16:33:18.081 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-10-15T16:33:18.092 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-10-15T16:33:18.211 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-10-15T16:33:18.234 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-10-15T16:33:18.338 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-10-15T16:33:18.382 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-10-15T16:33:18.468 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-10-15T16:33:18.533 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-10-15T16:33:18.596 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-10-15T16:33:18.670 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-10-15T16:33:18.734 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-10-15T16:33:18.818 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-10-15T16:33:18.861 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-10-15T16:33:18.946 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-10-15T16:33:18.999 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-10-15T16:33:19.095 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-10-15T16:33:19.127 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-10-15T16:33:19.222 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-10-15T16:33:19.254 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-10-15T16:33:19.358 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-10-15T16:33:19.380 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-10-15T16:33:19.486 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-10-15T16:33:19.508 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-10-15T16:33:19.624 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-10-15T16:33:19.637 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-10-15T16:33:19.763 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-10-15T16:33:19.764 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-10-15T16:33:19.891 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-10-15T16:33:19.913 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-10-15T16:33:20.017 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-10-15T16:33:20.039 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-10-15T16:33:20.153 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-10-15T16:33:20.175 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-10-15T16:33:20.280 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-10-15T16:33:20.301 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-10-15T16:33:20.418 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-10-15T16:33:20.451 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-10-15T16:33:20.546 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-10-15T16:33:20.588 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-10-15T16:33:20.672 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-10-15T16:33:20.726 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-10-15T16:33:20.801 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-10-15T16:33:20.864 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-10-15T16:33:20.926 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-10-15T16:33:21.010 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-10-15T16:33:21.053 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-10-15T16:33:21.149 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-10-15T16:33:21.193 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-10-15T16:33:21.288 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-15T16:33:21.321 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-10-15T16:33:21.426 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-10-15T16:33:21.459 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-15T16:33:21.553 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-10-15T16:33:21.586 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-10-15T16:33:21.703 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-10-15T16:33:21.715 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-10-15T16:33:21.822 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-10-15T16:33:21.844 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-10-15T16:33:21.968 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-10-15T16:33:21.969 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-10-15T16:33:22.105 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-10-15T16:33:22.106 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-10-15T16:33:22.233 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-10-15T16:33:22.234 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-10-15T16:33:22.350 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-10-15T16:33:22.361 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-10-15T16:33:22.486 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-10-15T16:33:22.489 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-10-15T16:33:22.625 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-10-15T16:33:22.627 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-10-15T16:33:22.751 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-10-15T16:33:22.752 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-10-15T16:33:22.878 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-10-15T16:33:22.889 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-10-15T16:33:23.014 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-10-15T16:33:23.015 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-10-15T16:33:23.141 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-10-15T16:33:23.143 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-10-15T16:33:23.280 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-10-15T16:33:23.281 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-10-15T16:33:23.407 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-10-15T16:33:23.408 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-10-15T16:33:23.534 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-10-15T16:33:23.536 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-10-15T16:33:23.660 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-10-15T16:33:23.671 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-10-15T16:33:23.786 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-10-15T16:33:23.808 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-10-15T16:33:23.913 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-10-15T16:33:23.945 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-10-15T16:33:24.050 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-10-15T16:33:24.073 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-10-15T16:33:24.178 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-10-15T16:33:24.211 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-10-15T16:33:24.315 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-10-15T16:33:24.358 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2020-10-15T16:33:24.441 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2020-10-15T16:33:24.484 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2020-10-15T16:33:24.568 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2020-10-15T16:33:24.621 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2020-10-15T16:33:24.694 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2020-10-15T16:33:24.757 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2020-10-15T16:33:24.831 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2020-10-15T16:33:24.894 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2020-10-15T16:33:24.957 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2020-10-15T16:33:25.040 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2020-10-15T16:33:25.117 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2020-10-15T16:33:25.213 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:25 +0000 (0:00:17.177) 0:00:25.172 ****** 2020-10-15T16:33:25.247 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:25 +0000 (0:00:00.033) 0:00:25.205 ****** 2020-10-15T16:33:25.253 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-10-15T16:33:25.291 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:25 +0000 (0:00:00.044) 0:00:25.249 ****** 2020-10-15T16:33:25.334 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:25 +0000 (0:00:00.043) 0:00:25.293 ****** 2020-10-15T16:33:25.343 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-10-15T16:33:25.345 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-10-15T16:33:25.346 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2020-10-15T16:33:25.348 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivanci 2020-10-15T16:33:25.349 INFO:teuthology.task.ansible.out:ch", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0Wihhvk 2020-10-15T16:33:25.351 INFO:teuthology.task.ansible.out:KztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-10-15T16:33:25.352 INFO:teuthology.task.ansible.out: HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2020-10-15T16:33:25.354 INFO:teuthology.task.ansible.out: e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2d 2020-10-15T16:33:25.355 INFO:teuthology.task.ansible.out:f79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx 2020-10-15T16:33:25.357 INFO:teuthology.task.ansible.out:+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4 2020-10-15T16:33:25.358 INFO:teuthology.task.ansible.out:b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq 2020-10-15T16:33:25.360 INFO:teuthology.task.ansible.out:/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelop 2020-10-15T16:33:25.361 INFO:teuthology.task.ansible.out:ez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}]}, "changed": false} 2020-10-15T16:33:25.403 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:25 +0000 (0:00:00.068) 0:00:25.362 ****** 2020-10-15T16:33:25.414 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-10-15T16:33:25.416 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2020-10-15T16:33:25.417 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivanci 2020-10-15T16:33:25.419 INFO:teuthology.task.ansible.out:ch", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0Wihhvk 2020-10-15T16:33:25.420 INFO:teuthology.task.ansible.out:KztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-10-15T16:33:25.421 INFO:teuthology.task.ansible.out: HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2020-10-15T16:33:25.423 INFO:teuthology.task.ansible.out: e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1 2020-10-15T16:33:25.423 INFO:teuthology.task.ansible.out:170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2d 2020-10-15T16:33:25.425 INFO:teuthology.task.ansible.out:f79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx 2020-10-15T16:33:25.426 INFO:teuthology.task.ansible.out:+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4 2020-10-15T16:33:25.428 INFO:teuthology.task.ansible.out:b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq 2020-10-15T16:33:25.429 INFO:teuthology.task.ansible.out:/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelop 2020-10-15T16:33:25.431 INFO:teuthology.task.ansible.out:ez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}]}, "changed": false} 2020-10-15T16:33:25.464 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:25 +0000 (0:00:00.060) 0:00:25.422 ****** 2020-10-15T16:33:26.214 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Clone the keys repo (5 retries left). 2020-10-15T16:33:26.227 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-10-15T16:33:26.228 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com -> localhost] => {"after": "8b571645c4accd90b6fb75de5ddc6e40874f17e2", "attempts": 1, "before": "8b571645c4accd90b6fb75de5ddc6e40874f17e2", "changed": false, "remote_url_changed": false} 2020-10-15T16:33:26.270 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:26 +0000 (0:00:00.806) 0:00:26.229 ****** 2020-10-15T16:33:26.477 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:26.478 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-10-15T16:33:26.614 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-10-15T16:33:26.760 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-10-15T16:33:26.895 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-10-15T16:33:26.897 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-10-15T16:33:27.010 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-10-15T16:33:27.155 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-10-15T16:33:27.157 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-10-15T16:33:27.269 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-10-15T16:33:27.414 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-10-15T16:33:27.529 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-10-15T16:33:27.540 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:27.685 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-10-15T16:33:27.687 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-10-15T16:33:27.810 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-10-15T16:33:27.812 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:27.821 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:27.945 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-10-15T16:33:28.080 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-10-15T16:33:28.082 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-10-15T16:33:28.205 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-10-15T16:33:28.332 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-10-15T16:33:28.468 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-10-15T16:33:28.593 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-10-15T16:33:28.718 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-10-15T16:33:28.855 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-10-15T16:33:28.980 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-10-15T16:33:28.982 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-10-15T16:33:29.106 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-10-15T16:33:29.107 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-10-15T16:33:29.210 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-10-15T16:33:29.325 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-10-15T16:33:29.471 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-10-15T16:33:29.473 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-10-15T16:33:29.597 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-10-15T16:33:29.712 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-10-15T16:33:29.856 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-10-15T16:33:29.991 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-10-15T16:33:30.126 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-10-15T16:33:30.255 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-10-15T16:33:30.257 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-10-15T16:33:30.380 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-10-15T16:33:30.496 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-10-15T16:33:30.611 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-10-15T16:33:30.725 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-10-15T16:33:30.849 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-10-15T16:33:30.984 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-10-15T16:33:31.120 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-15T16:33:31.246 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-10-15T16:33:31.360 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-10-15T16:33:31.496 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-10-15T16:33:31.497 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-10-15T16:33:31.621 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-10-15T16:33:31.623 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-10-15T16:33:31.725 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-10-15T16:33:31.851 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-10-15T16:33:31.996 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-10-15T16:33:32.121 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-10-15T16:33:32.123 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-15T16:33:32.261 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-10-15T16:33:32.263 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-10-15T16:33:32.264 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-10-15T16:33:32.387 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-10-15T16:33:32.502 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-10-15T16:33:32.617 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-10-15T16:33:32.731 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-10-15T16:33:32.876 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-10-15T16:33:33.001 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-10-15T16:33:33.003 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-10-15T16:33:33.136 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-10-15T16:33:33.272 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-10-15T16:33:33.396 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-10-15T16:33:33.398 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-10-15T16:33:33.512 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-10-15T16:33:33.660 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-10-15T16:33:33.775 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-10-15T16:33:33.776 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-10-15T16:33:33.921 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-10-15T16:33:34.035 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-10-15T16:33:34.183 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-10-15T16:33:34.318 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-15T16:33:34.453 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-10-15T16:33:34.600 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-10-15T16:33:34.734 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-10-15T16:33:34.860 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-10-15T16:33:34.996 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-10-15T16:33:35.140 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-10-15T16:33:35.276 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-10-15T16:33:35.421 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-10-15T16:33:35.557 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-10-15T16:33:35.691 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-10-15T16:33:35.826 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-10-15T16:33:35.952 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-10-15T16:33:36.076 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-10-15T16:33:36.211 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-10-15T16:33:36.371 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-10-15T16:33:36.506 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-10-15T16:33:36.632 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2020-10-15T16:33:36.633 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-10-15T16:33:36.756 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-10-15T16:33:36.778 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-10-15T16:33:36.780 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com -> localhost] => {"after": "8b571645c4accd90b6fb75de5ddc6e40874f17e2", "attempts": 2, "before": "8b571645c4accd90b6fb75de5ddc6e40874f17e2", "changed": false, "remote_url_changed": false} 2020-10-15T16:33:36.812 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:36 +0000 (0:00:10.541) 0:00:36.770 ****** 2020-10-15T16:33:36.901 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:36.903 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-10-15T16:33:36.985 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-10-15T16:33:37.037 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-10-15T16:33:37.120 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-10-15T16:33:37.152 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-10-15T16:33:37.236 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-10-15T16:33:37.278 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-10-15T16:33:37.280 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-10-15T16:33:37.382 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-10-15T16:33:37.384 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-10-15T16:33:37.425 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-10-15T16:33:37.427 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-10-15T16:33:37.499 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-10-15T16:33:37.562 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-10-15T16:33:37.636 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-10-15T16:33:37.638 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-10-15T16:33:37.699 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-10-15T16:33:37.761 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-10-15T16:33:37.835 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-10-15T16:33:37.887 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-10-15T16:33:37.970 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-10-15T16:33:38.013 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-10-15T16:33:38.014 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:38.086 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-10-15T16:33:38.139 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-10-15T16:33:38.141 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-10-15T16:33:38.212 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-10-15T16:33:38.265 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-10-15T16:33:38.267 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2020-10-15T16:33:38.276 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:38.287 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:38.331 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-10-15T16:33:38.415 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-10-15T16:33:38.479 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2020-10-15T16:33:38.481 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-10-15T16:33:38.532 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-10-15T16:33:38.534 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-10-15T16:33:38.605 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-10-15T16:33:38.668 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-10-15T16:33:38.731 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-10-15T16:33:38.794 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-10-15T16:33:38.879 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-10-15T16:33:38.923 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-10-15T16:33:38.997 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-10-15T16:33:39.040 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-10-15T16:33:39.133 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-10-15T16:33:39.166 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-10-15T16:33:39.261 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-10-15T16:33:39.293 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-10-15T16:33:39.378 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-10-15T16:33:39.380 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-15T16:33:39.421 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-10-15T16:33:39.423 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-10-15T16:33:39.526 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-10-15T16:33:39.548 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-10-15T16:33:39.550 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-10-15T16:33:39.663 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-10-15T16:33:39.685 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-10-15T16:33:39.789 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-10-15T16:33:39.810 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-10-15T16:33:39.914 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-10-15T16:33:39.936 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-10-15T16:33:39.938 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-10-15T16:33:40.040 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-10-15T16:33:40.061 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-10-15T16:33:40.175 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-10-15T16:33:40.187 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-10-15T16:33:40.307 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-10-15T16:33:40.329 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-10-15T16:33:40.433 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-10-15T16:33:40.455 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-10-15T16:33:40.570 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-10-15T16:33:40.581 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-10-15T16:33:40.685 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-10-15T16:33:40.716 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-10-15T16:33:40.718 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-10-15T16:33:40.821 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-10-15T16:33:40.852 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-10-15T16:33:40.937 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-10-15T16:33:40.969 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-10-15T16:33:41.072 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-10-15T16:33:41.094 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-10-15T16:33:41.188 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-10-15T16:33:41.225 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-10-15T16:33:41.330 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-10-15T16:33:41.351 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-10-15T16:33:41.475 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-10-15T16:33:41.477 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-10-15T16:33:41.600 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-15T16:33:41.611 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-10-15T16:33:41.716 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-10-15T16:33:41.737 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-10-15T16:33:41.851 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-10-15T16:33:41.873 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-10-15T16:33:41.875 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-10-15T16:33:41.978 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-10-15T16:33:41.980 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-10-15T16:33:41.999 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-10-15T16:33:42.103 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-10-15T16:33:42.105 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-10-15T16:33:42.124 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-10-15T16:33:42.229 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-10-15T16:33:42.271 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-10-15T16:33:42.355 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-10-15T16:33:42.387 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-10-15T16:33:42.480 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-10-15T16:33:42.522 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-10-15T16:33:42.605 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-10-15T16:33:42.607 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-15T16:33:42.668 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2020-10-15T16:33:42.741 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-10-15T16:33:42.743 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-10-15T16:33:42.745 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-10-15T16:33:42.794 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2020-10-15T16:33:42.856 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-10-15T16:33:42.929 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2020-10-15T16:33:42.982 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-10-15T16:33:43.055 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCavoAU8IGVih0HbUrIr5K44uQLdAq3r9MhKqnu754H8sigfFkxWPC6CsyKaGe0WnGcXz8wUdONqqew4WVvMN6KqE/8oLBf10yQYiigPw6ITPajrpj1JSsoW8kNIsfvrt+lzTUMWx0b6ThtQdANAu4OFZzERU//6yCy2Az0O0Ys9p4smgkwaXOA26hA48cxi147fBVC0Wl5xHuELKbmZ7pZFF/eAB9RLNlIRhBitWo93cLaWojivaJoxVhmKFgybuy7K+w1Tg7ACEJsSKXesO9AkcbFOXBuEwJQZDwyD4R6FfchloOGLBgb6Vn5bLDShWAgKlHE7eWa1EYG8qAjT9yeJEWIcQLz+8e3izFwriJAAzOqEB9a2GqVN6LQJzXTDcchJkOhhjfXaK3U3fqEfBozRtKVAUoNUgnlhYRbEcTMEuvz5AAUFB+0kvQpiw+RtieZUHl76KsNmFSIu7HSdgukdlkrGwOsl93qmYX96Q5AvanaAFj8ohbWSJSxZhKIB4U= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2020-10-15T16:33:43.130 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:43 +0000 (0:00:06.318) 0:00:43.088 ****** 2020-10-15T16:33:43.139 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-10-15T16:33:43.164 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:43.165 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.175 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.186 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.188 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.197 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.199 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.209 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.219 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.221 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.231 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:43.233 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-10-15T16:33:43.366 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-10-15T16:33:43.378 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:43.379 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2020-10-15T16:33:43.381 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.389 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.494 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:43.495 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-10-15T16:33:43.497 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-10-15T16:33:43.515 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:43.517 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-10-15T16:33:43.620 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-10-15T16:33:43.641 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:43.643 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-10-15T16:33:43.653 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.665 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.666 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.676 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.687 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.688 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.698 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.699 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.709 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.719 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.721 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.731 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.741 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.743 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:43.744 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.754 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.755 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.766 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.776 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.777 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.788 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.788 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.799 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.810 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.811 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.822 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.824 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.833 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.844 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.846 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.855 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.857 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.866 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.877 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.879 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.s 2020-10-15T16:33:43.880 INFO:teuthology.task.ansible.out:sh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gabrioux", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.889 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.890 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.901 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.911 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.913 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.923 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.923 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.934 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.945 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.946 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.956 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.967 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.969 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.978 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:43.989 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.000 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.001 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.012 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.023 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.024 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.034 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.035 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.045 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.057 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.059 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.068 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.079 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.080 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.090 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.091 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.101 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.112 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.113 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.123 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.123 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.124 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-10-15T16:33:44.124 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.135 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.137 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.146 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.157 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.159 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.168 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.170 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.180 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.191 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.192 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.202 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.213 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.214 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.224 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.235 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.236 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.246 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.247 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHIT 2020-10-15T16:33:44.249 INFO:teuthology.task.ansible.out:nmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-10-15T16:33:44.258 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:44.259 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.269 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.270 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.280 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.281 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.291 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.301 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.303 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.313 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.314 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.324 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.334 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.336 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.345 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.347 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.368 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.379 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.380 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.400 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.411 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.413 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.422 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.424 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.433 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.444 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.445 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.455 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.457 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.466 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.477 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.479 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.524 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:44 +0000 (0:00:01.394) 0:00:44.483 ****** 2020-10-15T16:33:44.532 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.533 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-10-15T16:33:44.544 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-10-15T16:33:44.546 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:44.577 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:44 +0000 (0:00:00.052) 0:00:44.535 ****** 2020-10-15T16:33:44.624 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.626 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-10-15T16:33:44.728 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:44.730 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-10-15T16:33:44.750 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.752 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-15T16:33:44.865 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:44.867 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-10-15T16:33:44.876 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.878 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-10-15T16:33:44.981 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:44.983 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-10-15T16:33:44.992 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:44.994 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-10-15T16:33:45.108 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.109 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-10-15T16:33:45.129 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:45.131 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-10-15T16:33:45.234 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.235 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-10-15T16:33:45.245 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:45.246 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-10-15T16:33:45.370 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.372 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-10-15T16:33:45.498 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.499 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-10-15T16:33:45.625 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-10-15T16:33:45.635 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.637 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-10-15T16:33:45.750 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:45.751 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-10-15T16:33:45.771 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.773 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2020-10-15T16:33:45.875 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:45.877 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-10-15T16:33:45.887 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:45.888 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-10-15T16:33:46.001 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.003 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-10-15T16:33:46.013 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.014 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-10-15T16:33:46.127 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.128 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-10-15T16:33:46.138 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.140 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-10-15T16:33:46.254 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.256 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-10-15T16:33:46.276 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.277 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-10-15T16:33:46.383 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.384 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-10-15T16:33:46.394 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.396 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-10-15T16:33:46.508 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.510 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-10-15T16:33:46.530 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.532 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-10-15T16:33:46.635 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.636 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-10-15T16:33:46.656 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.658 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-10-15T16:33:46.760 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.762 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-10-15T16:33:46.785 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:46.787 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-10-15T16:33:46.889 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:46.890 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2020-10-15T16:33:46.892 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi183.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-10-15T16:33:47.016 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-10-15T16:33:47.018 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-10-15T16:33:47.121 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.123 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-10-15T16:33:47.143 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.145 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-10-15T16:33:47.248 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.250 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-10-15T16:33:47.269 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.271 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-10-15T16:33:47.364 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.366 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-10-15T16:33:47.396 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.398 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-10-15T16:33:47.501 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.503 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-10-15T16:33:47.522 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.524 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-10-15T16:33:47.526 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-10-15T16:33:47.626 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.628 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-10-15T16:33:47.648 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.650 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-10-15T16:33:47.651 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-10-15T16:33:47.752 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.754 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-10-15T16:33:47.774 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.775 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-10-15T16:33:47.878 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.880 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-10-15T16:33:47.900 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:47.901 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-10-15T16:33:47.995 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:47.997 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-10-15T16:33:48.027 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.029 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-10-15T16:33:48.111 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi183.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-10-15T16:33:48.143 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.145 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-10-15T16:33:48.226 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.228 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-10-15T16:33:48.268 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.270 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-10-15T16:33:48.362 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.364 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-10-15T16:33:48.399 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.401 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-10-15T16:33:48.493 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.495 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-10-15T16:33:48.535 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.537 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-10-15T16:33:48.599 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.601 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-10-15T16:33:48.662 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.663 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2020-10-15T16:33:48.665 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-10-15T16:33:48.735 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.736 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-10-15T16:33:48.787 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.789 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-10-15T16:33:48.850 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.851 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-10-15T16:33:48.913 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:48.914 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-10-15T16:33:48.965 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:48.967 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-10-15T16:33:49.039 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.041 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-10-15T16:33:49.071 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:49.073 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-10-15T16:33:49.165 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.166 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-10-15T16:33:49.188 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:49.189 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2020-10-15T16:33:49.294 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-10-15T16:33:49.306 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:49.307 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-10-15T16:33:49.420 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.422 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-10-15T16:33:49.432 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:49.433 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-10-15T16:33:49.546 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.548 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-10-15T16:33:49.550 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi183.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-10-15T16:33:49.662 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-10-15T16:33:49.675 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.676 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-10-15T16:33:49.789 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:49.790 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-10-15T16:33:49.800 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.801 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-10-15T16:33:49.914 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:49.915 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-10-15T16:33:49.925 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:49.926 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-10-15T16:33:50.040 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:50.041 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-10-15T16:33:50.051 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:50.053 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-10-15T16:33:50.176 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:50.178 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-10-15T16:33:50.303 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:50.305 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-10-15T16:33:50.430 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-10-15T16:33:50.441 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:50.443 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2020-10-15T16:33:50.502 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:50 +0000 (0:00:05.925) 0:00:50.460 ****** 2020-10-15T16:33:50.560 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:50.562 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-10-15T16:33:50.686 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-10-15T16:33:50.811 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-10-15T16:33:50.833 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-10-15T16:33:50.835 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "checksum": "5e5fbdba250d2f3eb79c914bee029b0dace3cf55", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "e38c8e5e142876457f30007fef8740ab", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602779630.5229023-36275414048052/source", "state": "file", "uid": 0} 2020-10-15T16:33:50.867 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:50 +0000 (0:00:00.365) 0:00:50.826 ****** 2020-10-15T16:33:50.894 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-10-15T16:33:50.896 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => { "msg": "Host smithi183.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-10-15T16:33:50.929 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:50 +0000 (0:00:00.061) 0:00:50.887 ****** 2020-10-15T16:33:50.946 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:50.948 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-10-15T16:33:50.958 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-10-15T16:33:50.959 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-10-15T16:33:50.991 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:50 +0000 (0:00:00.062) 0:00:50.950 ****** 2020-10-15T16:33:51.017 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-10-15T16:33:51.019 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.050 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.058) 0:00:51.009 ****** 2020-10-15T16:33:51.066 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:51.068 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-10-15T16:33:51.078 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.110 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.060) 0:00:51.069 ****** 2020-10-15T16:33:51.137 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-10-15T16:33:51.139 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.170 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.059) 0:00:51.129 ****** 2020-10-15T16:33:51.187 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:51.188 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-10-15T16:33:51.198 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-10-15T16:33:51.199 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.235 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.064) 0:00:51.193 ****** 2020-10-15T16:33:51.262 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-10-15T16:33:51.263 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.295 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.059) 0:00:51.253 ****** 2020-10-15T16:33:51.311 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:51.313 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-10-15T16:33:51.323 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.356 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.061) 0:00:51.314 ****** 2020-10-15T16:33:51.383 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.415 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.059) 0:00:51.374 ****** 2020-10-15T16:33:51.432 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:51.434 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-10-15T16:33:51.443 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-10-15T16:33:51.445 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.479 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.063) 0:00:51.437 ****** 2020-10-15T16:33:51.505 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-10-15T16:33:51.507 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:33:51.539 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.060) 0:00:51.497 ****** 2020-10-15T16:33:51.566 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:51.567 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} TASK [common : Set have_entitlements] ****************************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.600 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.061) 0:00:51.559 ****** 2020-10-15T16:33:51.627 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-10-15T16:33:51.629 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.661 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.060) 0:00:51.619 ****** 2020-10-15T16:33:51.688 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-10-15T16:33:51.689 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-10-15T16:33:51.731 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.070) 0:00:51.690 ****** 2020-10-15T16:33:51.759 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.793 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.061) 0:00:51.751 ****** 2020-10-15T16:33:51.820 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:51.821 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} TASK [common : Register with subscription-manager.] **************************** skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:33:51.854 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.060) 0:00:51.812 ****** 2020-10-15T16:33:51.881 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-10-15T16:33:51.882 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:51.914 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.060) 0:00:51.872 ****** 2020-10-15T16:33:51.942 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-10-15T16:33:51.943 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4 2020-10-15T16:33:51.945 INFO:teuthology.task.ansible.out:yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-10-15T16:33:51.979 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:51 +0000 (0:00:00.064) 0:00:51.937 ****** 2020-10-15T16:33:52.006 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-10-15T16:33:52.008 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.040 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.060) 0:00:51.998 ****** 2020-10-15T16:33:52.067 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-10-15T16:33:52.069 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.101 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.061) 0:00:52.059 ****** 2020-10-15T16:33:52.108 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.110 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-10-15T16:33:52.130 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-10-15T16:33:52.132 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:33:52.163 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.062) 0:00:52.122 ****** 2020-10-15T16:33:52.190 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-15T16:33:52.192 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.226 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.062) 0:00:52.185 ****** 2020-10-15T16:33:52.232 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.234 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-10-15T16:33:52.255 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.288 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.061) 0:00:52.246 ****** 2020-10-15T16:33:52.317 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-10-15T16:33:52.318 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.352 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.064) 0:00:52.310 ****** 2020-10-15T16:33:52.361 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.363 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-10-15T16:33:52.384 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-10-15T16:33:52.386 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.422 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.070) 0:00:52.381 ****** 2020-10-15T16:33:52.452 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-10-15T16:33:52.454 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.491 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.069) 0:00:52.450 ****** 2020-10-15T16:33:52.499 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.501 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-10-15T16:33:52.522 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-10-15T16:33:52.523 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.557 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.065) 0:00:52.516 ****** 2020-10-15T16:33:52.588 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-10-15T16:33:52.589 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-10-15T16:33:52.623 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.065) 0:00:52.581 ****** 2020-10-15T16:33:52.650 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-10-15T16:33:52.652 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.684 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.060) 0:00:52.642 ****** 2020-10-15T16:33:52.714 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.716 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.767 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.082) 0:00:52.725 ****** 2020-10-15T16:33:52.799 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-10-15T16:33:52.800 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.851 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.084) 0:00:52.809 ****** 2020-10-15T16:33:52.860 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.862 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2020-10-15T16:33:52.884 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-10-15T16:33:52.886 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.930 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.079) 0:00:52.889 ****** 2020-10-15T16:33:52.957 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:52.959 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi183.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-10-15T16:33:52.992 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:52 +0000 (0:00:00.061) 0:00:52.951 ****** 2020-10-15T16:33:53.019 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-15T16:33:53.020 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.054 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.061) 0:00:53.012 ****** 2020-10-15T16:33:53.091 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-15T16:33:53.093 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCavoAU8IGVih0HbUrIr5K44uQLdAq3r9MhKqnu754H8sigfFkxWPC6CsyKaGe0WnGcXz8wUdONqqew4WVvMN6KqE/8oLBf10yQYiigPw6ITPajrpj1JSsoW8kNIsfvrt+lzTUMWx0b6ThtQdANAu4OFZzERU//6yCy2Az0O0Ys9p4smgkwaXOA26hA48cxi147fBVC0Wl5xHuELKbmZ7pZFF/eAB9RLNlIRhBitWo93cLaWojivaJoxVhmKFgybuy7K+w1Tg7ACEJsSKXesO9AkcbFOXBuEwJQZDwyD4R6FfchloOGLBgb6Vn5bLDShWAgKlHE7eWa1EYG8qAjT9yeJEWIcQLz+8e3izFwriJAAzOqEB9a2GqVN6LQJzXTDcchJkOhhjfXaK3U3fqEfBozRtKVAUoNUgnlhYRbEcTMEuvz5AAUFB+0kvQpiw+RtieZUHl76KsNmFSIu7HSdgukdlkrGwOsl93qmYX96Q5AvanaAFj8ohbWSJSxZhKIB4U= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2020-10-15T16:33:53.162 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.108) 0:00:53.121 ****** 2020-10-15T16:33:53.201 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:53.202 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.212 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.020591", "end": "2020-10-15 16:33:53.354591", "rc": 0, "start": "2020-10-15 16:33:53.334000", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-10-15T16:33:53.214 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.247 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.084) 0:00:53.205 ****** 2020-10-15T16:33:53.254 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.256 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd 2020-10-15T16:33:53.258 INFO:teuthology.task.ansible.out:0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible 2020-10-15T16:33:53.259 INFO:teuthology.task.ansible.out:_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.279 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.312 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.065) 0:00:53.270 ****** 2020-10-15T16:33:53.341 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.373 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.061) 0:00:53.332 ****** 2020-10-15T16:33:53.390 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:53.392 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-10-15T16:33:53.401 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-10-15T16:33:53.403 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.436 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.062) 0:00:53.394 ****** 2020-10-15T16:33:53.463 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-10-15T16:33:53.464 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.496 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.060) 0:00:53.454 ****** 2020-10-15T16:33:53.525 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-10-15T16:33:53.526 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.557 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.061) 0:00:53.516 ****** 2020-10-15T16:33:53.564 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:53.565 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-10-15T16:33:53.586 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.619 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.061) 0:00:53.578 ****** 2020-10-15T16:33:53.649 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-10-15T16:33:53.651 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.685 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:53 +0000 (0:00:00.065) 0:00:53.643 ****** 2020-10-15T16:33:53.692 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:53.693 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2020-10-15T16:33:53.695 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@n 2020-10-15T16:33:53.696 INFO:teuthology.task.ansible.out:ew-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.707 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.708 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.718 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.729 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.731 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.741 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.752 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.753 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.763 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.765 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.775 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.786 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.787 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.798 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.800 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.810 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.821 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.822 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.833 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.835 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.845 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.846 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.857 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.867 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.868 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.879 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.880 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.891 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.901 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.903 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.913 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.924 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.925 INFO:teuthology.task.ansible.out: skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.936 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.937 INFO:teuthology.task.ansible.out: skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.948 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.949 INFO:teuthology.task.ansible.out: skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.959 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.961 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:53.971 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.973 INFO:teuthology.task.ansible.out: skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.983 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.984 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:53.994 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:53.996 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.005 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.016 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.017 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.028 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.029 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.039 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.050 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.051 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.062 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.073 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.075 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.085 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.086 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.096 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.107 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.109 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.119 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.121 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.130 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.141 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.152 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.154 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.164 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.175 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.177 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.186 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.197 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.199 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.208 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.219 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.221 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.231 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.242 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.244 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.253 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.255 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.265 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.276 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.278 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.287 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.298 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.300 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.309 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.320 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.322 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.331 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.342 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.344 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.353 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.355 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.365 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-10-15T16:33:54.366 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.554115", "end": "2020-10-15 16:33:54.519170", "rc": 0, "start": "2020-10-15 16:33:53.965055", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.399 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:54 +0000 (0:00:00.714) 0:00:54.358 ****** 2020-10-15T16:33:54.406 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.408 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.409 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": 2020-10-15T16:33:54.410 INFO:teuthology.task.ansible.out: "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.420 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.421 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-10-15T16:33:54.421 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.459 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:54 +0000 (0:00:00.059) 0:00:54.418 ****** 2020-10-15T16:33:54.467 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-15T16:33:54.468 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.469 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.470 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.481 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.482 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.492 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.493 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.504 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.554 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:54 +0000 (0:00:00.094) 0:00:54.512 ****** 2020-10-15T16:33:54.582 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-10-15T16:33:54.584 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:54.615 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:54 +0000 (0:00:00.061) 0:00:54.574 ****** 2020-10-15T16:33:54.643 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-10-15T16:33:54.645 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.053008", "end": "2020-10-15 16:33:54.791956", "rc": 0, "start": "2020-10-15 16:33:54.738948", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:33:54.677 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:54 +0000 (0:00:00.062) 0:00:54.636 ****** 2020-10-15T16:33:54.777 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:54.779 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-10-15T16:33:54.903 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-10-15T16:33:55.028 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-10-15T16:33:55.142 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-10-15T16:33:55.267 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-10-15T16:33:55.394 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-10-15T16:33:55.518 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-10-15T16:33:55.634 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-10-15T16:33:55.759 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2020-10-15T16:33:55.760 INFO:teuthology.task.ansible.out: 2020-10-15T16:33:55.884 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-10-15T16:33:56.009 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-10-15T16:33:56.125 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-10-15T16:33:56.252 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-10-15T16:33:56.378 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-10-15T16:33:56.504 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-10-15T16:33:56.619 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-10-15T16:33:56.744 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-10-15T16:33:56.868 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-10-15T16:33:56.982 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-10-15T16:33:57.107 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-10-15T16:33:57.232 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-10-15T16:33:57.346 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-10-15T16:33:57.472 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-10-15T16:33:57.597 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-10-15T16:33:57.723 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-10-15T16:33:57.838 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-10-15T16:33:57.962 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-10-15T16:33:58.087 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-10-15T16:33:58.202 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-10-15T16:33:58.329 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-10-15T16:33:58.455 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-10-15T16:33:58.510 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-10-15T16:33:58.512 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:33:58.553 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:58 +0000 (0:00:03.875) 0:00:58.511 ****** 2020-10-15T16:33:58.570 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:58.571 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-10-15T16:33:58.696 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-10-15T16:33:58.822 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-10-15T16:33:58.937 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-10-15T16:33:59.063 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-10-15T16:33:59.180 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2020-10-15T16:33:59.305 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-10-15T16:33:59.409 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-10-15T16:33:59.411 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:33:59.442 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:59 +0000 (0:00:00.889) 0:00:59.401 ****** 2020-10-15T16:33:59.448 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:59.450 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-10-15T16:33:59.471 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:33:59.503 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:33:59 +0000 (0:00:00.061) 0:00:59.462 ****** 2020-10-15T16:33:59.562 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:33:59.564 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-10-15T16:33:59.678 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-10-15T16:33:59.803 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-10-15T16:33:59.927 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-10-15T16:34:00.042 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-10-15T16:34:00.104 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-10-15T16:34:00.106 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-10-15T16:34:00.138 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:00 +0000 (0:00:00.634) 0:01:00.096 ****** 2020-10-15T16:34:00.165 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-15T16:34:00.167 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-10-15T16:34:00.280 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2020-10-15T16:34:00.405 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2020-10-15T16:34:00.456 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:00 +0000 (0:00:00.318) 0:01:00.415 ****** 2020-10-15T16:34:00.807 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-10-15T16:34:00.809 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "checksum": "5e5fbdba250d2f3eb79c914bee029b0dace3cf55", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "e38c8e5e142876457f30007fef8740ab", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602779640.484781-1571372980698/source", "state": "file", "uid": 0} 2020-10-15T16:34:00.840 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:00 +0000 (0:00:00.383) 0:01:00.799 ****** 2020-10-15T16:34:00.869 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi038.front.sepia.ceph.com] => { "msg": "Host smithi038.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-10-15T16:34:00.901 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:00 +0000 (0:00:00.061) 0:01:00.860 ****** 2020-10-15T16:34:00.931 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-10-15T16:34:00.932 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-10-15T16:34:00.964 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:00 +0000 (0:00:00.062) 0:01:00.923 ****** 2020-10-15T16:34:00.971 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-10-15T16:34:00.973 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:34:01.005 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:00 +0000 (0:00:00.040) 0:01:00.964 ****** 2020-10-15T16:34:01.012 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-10-15T16:34:01.014 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.047 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.041) 0:01:01.005 ****** 2020-10-15T16:34:01.073 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-10-15T16:34:01.075 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.106 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.059) 0:01:01.065 ****** 2020-10-15T16:34:01.133 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-10-15T16:34:01.135 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.166 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.059) 0:01:01.124 ****** 2020-10-15T16:34:01.196 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-10-15T16:34:01.197 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.228 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.062) 0:01:01.186 ****** 2020-10-15T16:34:01.258 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-10-15T16:34:01.260 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.291 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.063) 0:01:01.250 ****** 2020-10-15T16:34:01.319 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-10-15T16:34:01.321 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.352 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.060) 0:01:01.310 ****** 2020-10-15T16:34:01.379 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-10-15T16:34:01.381 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.412 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.060) 0:01:01.371 ****** 2020-10-15T16:34:01.440 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-10-15T16:34:01.441 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.473 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.060) 0:01:01.431 ****** 2020-10-15T16:34:01.480 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-15T16:34:01.482 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-10-15T16:34:01.503 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-10-15T16:34:01.504 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:01.536 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.062) 0:01:01.494 ****** 2020-10-15T16:34:01.566 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-10-15T16:34:01.568 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.599 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.063) 0:01:01.558 ****** 2020-10-15T16:34:01.627 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-10-15T16:34:01.629 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.660 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.060) 0:01:01.618 ****** 2020-10-15T16:34:01.688 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-10-15T16:34:01.690 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:01.721 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.060) 0:01:01.679 ****** 2020-10-15T16:34:01.749 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-10-15T16:34:01.750 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.782 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.061) 0:01:01.741 ****** 2020-10-15T16:34:01.789 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-15T16:34:01.791 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-10-15T16:34:01.801 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-10-15T16:34:01.803 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:01.834 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.051) 0:01:01.793 ****** 2020-10-15T16:34:01.862 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-10-15T16:34:01.864 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.894 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.060) 0:01:01.853 ****** 2020-10-15T16:34:01.925 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-10-15T16:34:01.926 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:01.957 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:01 +0000 (0:00:00.062) 0:01:01.916 ****** 2020-10-15T16:34:01.985 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-10-15T16:34:01.987 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.018 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.060) 0:01:01.976 ****** 2020-10-15T16:34:02.045 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-10-15T16:34:02.046 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.078 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.060) 0:01:02.036 ****** 2020-10-15T16:34:02.085 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-15T16:34:02.086 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-10-15T16:34:02.121 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.043) 0:01:02.079 ****** 2020-10-15T16:34:02.129 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-10-15T16:34:02.130 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:02.164 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.043) 0:01:02.122 ****** 2020-10-15T16:34:02.192 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-15T16:34:02.193 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.226 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.061) 0:01:02.184 ****** 2020-10-15T16:34:02.244 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-10-15T16:34:02.246 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-10-15T16:34:02.278 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.051) 0:01:02.236 ****** 2020-10-15T16:34:02.284 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-10-15T16:34:02.286 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.319 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.040) 0:01:02.277 ****** 2020-10-15T16:34:02.346 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-10-15T16:34:02.348 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.379 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.060) 0:01:02.338 ****** 2020-10-15T16:34:02.406 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-10-15T16:34:02.408 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.439 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.059) 0:01:02.398 ****** 2020-10-15T16:34:02.466 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-10-15T16:34:02.468 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.500 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.061) 0:01:02.459 ****** 2020-10-15T16:34:02.528 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-10-15T16:34:02.529 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.560 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.060) 0:01:02.519 ****** 2020-10-15T16:34:02.587 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-10-15T16:34:02.589 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:02.624 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.063) 0:01:02.582 ****** 2020-10-15T16:34:02.631 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-15T16:34:02.633 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-10-15T16:34:02.643 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-10-15T16:34:02.645 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.676 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.052) 0:01:02.635 ****** 2020-10-15T16:34:02.705 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-10-15T16:34:02.707 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.739 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.062) 0:01:02.697 ****** 2020-10-15T16:34:02.769 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-10-15T16:34:02.771 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.802 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.063) 0:01:02.761 ****** 2020-10-15T16:34:02.843 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-10-15T16:34:02.844 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.854 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.892 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.090) 0:01:02.851 ****** 2020-10-15T16:34:02.900 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-15T16:34:02.902 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-10-15T16:34:02.924 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-10-15T16:34:02.925 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.935 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:02.968 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:02 +0000 (0:00:00.076) 0:01:02.927 ****** 2020-10-15T16:34:02.996 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-15T16:34:02.996 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.030 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.061) 0:01:02.989 ****** 2020-10-15T16:34:03.142 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-15T16:34:03.143 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-10-15T16:34:03.178 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.148) 0:01:03.137 ****** 2020-10-15T16:34:03.186 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-10-15T16:34:03.187 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.016915", "end": "2020-10-15 16:34:02.890936", "rc": 0, "start": "2020-10-15 16:34:02.874021", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-10-15T16:34:03.220 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.041) 0:01:03.178 ****** 2020-10-15T16:34:03.228 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-10-15T16:34:03.229 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-10-15T16:34:03.261 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.041) 0:01:03.219 ****** 2020-10-15T16:34:03.266 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-10-15T16:34:03.268 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.301 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.040) 0:01:03.259 ****** 2020-10-15T16:34:03.307 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-10-15T16:34:03.308 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.341 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.040) 0:01:03.299 ****** 2020-10-15T16:34:03.347 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-10-15T16:34:03.349 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.382 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.040) 0:01:03.340 ****** 2020-10-15T16:34:03.388 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-10-15T16:34:03.389 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:03.424 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.042) 0:01:03.382 ****** 2020-10-15T16:34:03.430 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-10-15T16:34:03.432 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.465 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.041) 0:01:03.424 ****** 2020-10-15T16:34:03.472 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-10-15T16:34:03.473 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.474 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.485 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.486 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.518 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.053) 0:01:03.477 ****** 2020-10-15T16:34:03.549 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.030) 0:01:03.507 ****** 2020-10-15T16:34:03.554 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-10-15T16:34:03.556 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.588 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.039) 0:01:03.547 ****** 2020-10-15T16:34:03.595 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-10-15T16:34:03.597 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.628 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.039) 0:01:03.587 ****** 2020-10-15T16:34:03.634 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-10-15T16:34:03.636 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.669 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.041) 0:01:03.628 ****** 2020-10-15T16:34:03.696 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-10-15T16:34:03.698 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:03.729 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:03 +0000 (0:00:00.059) 0:01:03.688 ****** 2020-10-15T16:34:04.571 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-10-15T16:34:04.573 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:34:04.609 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:04 +0000 (0:00:00.879) 0:01:04.567 ****** 2020-10-15T16:34:04.802 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-10-15T16:34:04.803 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.947099", "end": "2020-10-15 16:34:04.509479", "rc": 0, "start": "2020-10-15 16:34:03.562380", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:04.836 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:04 +0000 (0:00:00.227) 0:01:04.795 ****** 2020-10-15T16:34:04.863 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-10-15T16:34:04.865 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:04.900 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:04 +0000 (0:00:00.063) 0:01:04.858 ****** 2020-10-15T16:34:04.916 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-10-15T16:34:04.918 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-10-15T16:34:04.951 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:04 +0000 (0:00:00.051) 0:01:04.910 ****** 2020-10-15T16:34:05.073 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-10-15T16:34:05.074 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.044097", "end": "2020-10-15 16:34:04.778171", "rc": 0, "start": "2020-10-15 16:34:04.734074", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:05.108 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:05 +0000 (0:00:00.156) 0:01:05.066 ****** 2020-10-15T16:34:05.259 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-10-15T16:34:05.261 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-10-15T16:34:05.294 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:05 +0000 (0:00:00.186) 0:01:05.253 ****** 2020-10-15T16:34:05.425 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-10-15T16:34:05.427 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-10-15T16:34:05.530 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-10-15T16:34:05.565 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:05 +0000 (0:00:00.270) 0:01:05.523 ****** 2020-10-15T16:34:05.592 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-10-15T16:34:05.593 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:05.627 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:05 +0000 (0:00:00.062) 0:01:05.585 ****** 2020-10-15T16:34:05.655 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-10-15T16:34:05.657 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:05.691 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:05 +0000 (0:00:00.063) 0:01:05.649 ****** 2020-10-15T16:34:05.718 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-10-15T16:34:05.719 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:05.756 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:05 +0000 (0:00:00.065) 0:01:05.715 ****** 2020-10-15T16:34:06.077 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-10-15T16:34:06.078 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-10-15T16:34:06.116 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:06 +0000 (0:00:00.360) 0:01:06.075 ****** 2020-10-15T16:34:06.765 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-10-15T16:34:06.766 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-10-15 16:32:52 UTC", "ActiveEnterTimestampMonotonic": "20406585", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target network.target var-run.mount sysinit.target basic.target local-fs.target systemd-tmpfiles-setup.service nss-lookup.target remote-fs.target systemd-journald.socket -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-10-15 16:32:52 UTC", "AssertTimestampMonotonic": "20405782", "Before": "plymouth-quit.service shutdown.target getty@tty1.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-10-15 16:32:52 UTC", "ConditionTimestampMonotonic": "20405782", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-10-15T16:34:06.767 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1290", "ExecMainStartTimestamp": "Thu 2020-10-15 16:32:52 UTC", "ExecMainStartTimestampMonotonic": "20406560", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-10-15 16:32:52 UTC] ; stop_time=[n/a] ; pid=1290 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-10-15 16:32:52 UTC", "InactiveExitTimestampMonotonic": "20406585", "InvocationID": "897880f9f56d4c8292e91fa99bdf381f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-10-15T16:34:06.769 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1290", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-10-15T16:34:06.770 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-10-15 16:32:52 UTC", "StateChangeTimestampMonotonic": "20406585", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-10-15 16:32:52 UTC", "WatchdogTimestampMonotonic": "20406584", "WatchdogUSec": "0"}} 2020-10-15T16:34:06.800 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:06 +0000 (0:00:00.683) 0:01:06.759 ****** 2020-10-15T16:34:06.827 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-10-15T16:34:06.829 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:06.861 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:06 +0000 (0:00:00.060) 0:01:06.819 ****** 2020-10-15T16:34:06.888 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-10-15T16:34:06.889 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:06.921 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:06 +0000 (0:00:00.059) 0:01:06.879 ****** 2020-10-15T16:34:06.947 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-10-15T16:34:06.949 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:06.981 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:06 +0000 (0:00:00.060) 0:01:06.939 ****** 2020-10-15T16:34:07.008 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-10-15T16:34:07.010 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.041 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.060) 0:01:07.000 ****** 2020-10-15T16:34:07.068 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-10-15T16:34:07.070 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.101 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.060) 0:01:07.060 ****** 2020-10-15T16:34:07.128 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-10-15T16:34:07.130 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.162 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.060) 0:01:07.120 ****** 2020-10-15T16:34:07.189 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-10-15T16:34:07.190 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.225 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.063) 0:01:07.183 ****** 2020-10-15T16:34:07.252 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-10-15T16:34:07.254 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.285 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.060) 0:01:07.243 ****** 2020-10-15T16:34:07.312 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-10-15T16:34:07.314 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.345 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.060) 0:01:07.303 ****** 2020-10-15T16:34:07.372 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-10-15T16:34:07.374 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:07.405 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.060) 0:01:07.364 ****** 2020-10-15T16:34:07.432 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-10-15T16:34:07.434 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-10-15T16:34:07.469 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.063) 0:01:07.427 ****** 2020-10-15T16:34:07.506 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-10-15T16:34:07.507 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-10-15T16:34:07.540 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.071) 0:01:07.499 ****** 2020-10-15T16:34:07.578 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-10-15T16:34:07.579 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-10-15T16:34:07.612 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.071) 0:01:07.570 ****** 2020-10-15T16:34:07.649 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-10-15T16:34:07.651 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-10-15T16:34:07.683 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.071) 0:01:07.642 ****** 2020-10-15T16:34:07.813 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-10-15T16:34:07.815 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-10-15T16:34:07.847 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:07 +0000 (0:00:00.163) 0:01:07.805 ****** 2020-10-15T16:34:07.978 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-10-15T16:34:07.979 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "name": "ubuntu", "state": "present", "system": false} 2020-10-15T16:34:08.011 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.164) 0:01:07.969 ****** 2020-10-15T16:34:08.038 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-10-15T16:34:08.039 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:08.070 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.059) 0:01:08.029 ****** 2020-10-15T16:34:08.098 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-10-15T16:34:08.099 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:08.132 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.061) 0:01:08.091 ****** 2020-10-15T16:34:08.284 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-10-15T16:34:08.286 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-10-15T16:34:08.317 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.185) 0:01:08.276 ****** 2020-10-15T16:34:08.688 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-10-15T16:34:08.690 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:08.722 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.404) 0:01:08.680 ****** 2020-10-15T16:34:08.729 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-10-15T16:34:08.731 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-10-15T16:34:08.765 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.043) 0:01:08.724 ****** 2020-10-15T16:34:08.809 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-10-15T16:34:08.811 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi183"}, "changed": false} 2020-10-15T16:34:08.842 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:08 +0000 (0:00:00.076) 0:01:08.801 ****** 2020-10-15T16:34:09.640 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-10-15T16:34:09.642 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:09.675 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:09 +0000 (0:00:00.832) 0:01:09.633 ****** 2020-10-15T16:34:09.703 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-10-15T16:34:09.705 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:09.738 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:09 +0000 (0:00:00.063) 0:01:09.697 ****** 2020-10-15T16:34:09.746 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-10-15T16:34:09.748 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi183.front.sepia.ceph.com", "ansible_hostname": "smithi183", "ansible_nodename": "smithi183"}, "changed": true, "name": "smithi183"} 2020-10-15T16:34:09.784 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:09 +0000 (0:00:00.045) 0:01:09.742 ****** 2020-10-15T16:34:10.040 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-10-15T16:34:10.042 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-10-15T16:34:10.077 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:10 +0000 (0:00:00.293) 0:01:10.035 ****** 2020-10-15T16:34:10.096 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-10-15T16:34:10.097 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-10-15T16:34:10.134 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:10 +0000 (0:00:00.056) 0:01:10.092 ****** 2020-10-15T16:34:10.453 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-10-15T16:34:10.455 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-10-15T16:34:10.488 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:10 +0000 (0:00:00.354) 0:01:10.446 ****** 2020-10-15T16:34:10.940 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-10-15T16:34:10.942 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:10.977 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:10 +0000 (0:00:00.489) 0:01:10.936 ****** 2020-10-15T16:34:10.995 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-10-15T16:34:10.997 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-10-15T16:34:11.030 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.052) 0:01:10.988 ****** 2020-10-15T16:34:11.058 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-10-15T16:34:11.060 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.092 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.061) 0:01:11.050 ****** 2020-10-15T16:34:11.120 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-15T16:34:11.122 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.153 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.061) 0:01:11.112 ****** 2020-10-15T16:34:11.181 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-15T16:34:11.183 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.214 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.060) 0:01:11.172 ****** 2020-10-15T16:34:11.241 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-10-15T16:34:11.243 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.279 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.065) 0:01:11.237 ****** 2020-10-15T16:34:11.307 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-15T16:34:11.309 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.341 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.062) 0:01:11.300 ****** 2020-10-15T16:34:11.369 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-15T16:34:11.371 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.402 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.060) 0:01:11.361 ****** 2020-10-15T16:34:11.433 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-15T16:34:11.435 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.470 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.067) 0:01:11.428 ****** 2020-10-15T16:34:11.508 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-10-15T16:34:11.511 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.542 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.071) 0:01:11.500 ****** 2020-10-15T16:34:11.570 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-10-15T16:34:11.604 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.062) 0:01:11.563 ****** 2020-10-15T16:34:11.611 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-15T16:34:11.613 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-10-15T16:34:11.634 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-10-15T16:34:11.676 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.071) 0:01:11.635 ****** 2020-10-15T16:34:11.704 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-10-15T16:34:11.741 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.064) 0:01:11.699 ****** 2020-10-15T16:34:11.768 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-10-15T16:34:11.770 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:11.924 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.183) 0:01:11.882 ****** 2020-10-15T16:34:11.931 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-15T16:34:11.933 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-10-15T16:34:11.972 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:11 +0000 (0:00:00.047) 0:01:11.930 ****** 2020-10-15T16:34:11.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-10-15T16:34:11.980 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.015 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.043) 0:01:11.974 ****** 2020-10-15T16:34:12.042 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.076 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.060) 0:01:12.034 ****** 2020-10-15T16:34:12.093 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-10-15T16:34:12.095 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-10-15T16:34:12.126 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.050) 0:01:12.084 ****** 2020-10-15T16:34:12.133 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:12.135 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.169 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.042) 0:01:12.127 ****** 2020-10-15T16:34:12.196 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.230 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.060) 0:01:12.188 ****** 2020-10-15T16:34:12.257 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-10-15T16:34:12.259 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.290 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.060) 0:01:12.249 ****** 2020-10-15T16:34:12.318 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-10-15T16:34:12.320 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.353 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.062) 0:01:12.312 ****** 2020-10-15T16:34:12.381 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-10-15T16:34:12.382 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.415 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.061) 0:01:12.373 ****** 2020-10-15T16:34:12.433 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-15T16:34:12.435 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-10-15T16:34:12.445 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-15T16:34:12.447 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.480 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.065) 0:01:12.439 ****** 2020-10-15T16:34:12.508 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-15T16:34:12.509 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.541 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.061) 0:01:12.500 ****** 2020-10-15T16:34:12.569 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-15T16:34:12.570 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.607 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.065) 0:01:12.566 ****** 2020-10-15T16:34:12.645 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-10-15T16:34:12.647 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.683 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.075) 0:01:12.641 ****** 2020-10-15T16:34:12.710 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-15T16:34:12.712 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi038.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-10-15T16:34:12.746 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.062) 0:01:12.704 ****** 2020-10-15T16:34:12.773 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-15T16:34:12.775 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.808 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.061) 0:01:12.766 ****** 2020-10-15T16:34:12.835 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-10-15T16:34:12.837 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.870 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.062) 0:01:12.829 ****** 2020-10-15T16:34:12.897 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-10-15T16:34:12.899 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.932 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.061) 0:01:12.890 ****** 2020-10-15T16:34:12.961 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-10-15T16:34:12.963 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:12.997 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:12 +0000 (0:00:00.065) 0:01:12.956 ****** 2020-10-15T16:34:13.004 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-15T16:34:13.006 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-10-15T16:34:13.048 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.050) 0:01:13.006 ****** 2020-10-15T16:34:13.059 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-10-15T16:34:13.061 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.097 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.049) 0:01:13.055 ****** 2020-10-15T16:34:13.105 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-10-15T16:34:13.107 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-10-15T16:34:13.160 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.062) 0:01:13.118 ****** 2020-10-15T16:34:13.170 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-10-15T16:34:13.171 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.208 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.048) 0:01:13.166 ****** 2020-10-15T16:34:13.214 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-10-15T16:34:13.216 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.251 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.042) 0:01:13.209 ****** 2020-10-15T16:34:13.258 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-10-15T16:34:13.260 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.294 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.043) 0:01:13.253 ****** 2020-10-15T16:34:13.301 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-10-15T16:34:13.303 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:13.338 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.043) 0:01:13.296 ****** 2020-10-15T16:34:13.345 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-10-15T16:34:13.347 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.382 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.044) 0:01:13.341 ****** 2020-10-15T16:34:13.395 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-10-15T16:34:13.396 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.397 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.408 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.410 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.456 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.073) 0:01:13.415 ****** 2020-10-15T16:34:13.463 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-10-15T16:34:13.465 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.504 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.047) 0:01:13.462 ****** 2020-10-15T16:34:13.546 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-10-15T16:34:13.547 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:13.596 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.092) 0:01:13.555 ****** 2020-10-15T16:34:13.615 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-10-15T16:34:13.617 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004540", "end": "2020-10-15 16:34:13.751187", "rc": 0, "start": "2020-10-15 16:34:13.746647", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:13.650 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.054) 0:01:13.609 ****** 2020-10-15T16:34:13.678 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-10-15T16:34:13.712 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:13 +0000 (0:00:00.061) 0:01:13.670 ****** 2020-10-15T16:34:14.040 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-10-15T16:34:14.041 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-10-15T16:34:14.074 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:14 +0000 (0:00:00.362) 0:01:14.033 ****** 2020-10-15T16:34:14.102 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-10-15T16:34:14.104 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:14.136 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:14 +0000 (0:00:00.061) 0:01:14.094 ****** 2020-10-15T16:34:14.514 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-10-15T16:34:14.516 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:14.553 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:14 +0000 (0:00:00.416) 0:01:14.511 ****** 2020-10-15T16:34:14.866 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-10-15T16:34:14.868 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-10-15T16:34:14.899 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:14 +0000 (0:00:00.346) 0:01:14.858 ****** 2020-10-15T16:34:15.201 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-10-15T16:34:15.204 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-10-15T16:34:15.234 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:15 +0000 (0:00:00.334) 0:01:15.192 ****** 2020-10-15T16:34:15.345 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-10-15T16:34:15.347 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-10-15T16:34:15.377 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-10-15T16:34:15.379 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-10-15T16:34:15.483 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-10-15T16:34:15.517 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:15 +0000 (0:00:00.283) 0:01:15.476 ****** 2020-10-15T16:34:15.544 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-10-15T16:34:15.546 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:15.577 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:15 +0000 (0:00:00.059) 0:01:15.535 ****** 2020-10-15T16:34:15.604 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-10-15T16:34:15.606 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-15T16:34:15.638 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:15 +0000 (0:00:00.061) 0:01:15.597 ****** 2020-10-15T16:34:15.665 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-10-15T16:34:15.667 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:15.699 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:15 +0000 (0:00:00.060) 0:01:15.657 ****** 2020-10-15T16:34:16.026 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-10-15T16:34:16.028 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-10-15T16:34:16.060 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.360) 0:01:16.018 ****** 2020-10-15T16:34:16.265 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-10-15T16:34:16.266 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-10-15 16:32:47 UTC", "ActiveEnterTimestampMonotonic": "17810966", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount sysinit.target -.mount nss-lookup.target network.target local-fs.target basic.target systemd-tmpfiles-setup.service system.slice systemd-journald.socket time-sync.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-10-15 16:32:47 UTC", "AssertTimestampMonotonic": "17810242", "Before": "xdm.service multi-user.target plymouth-quit.service shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-10-15 16:32:47 UTC", "ConditionTimestampMonotonic": "17810241", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-10-15T16:34:16.268 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1347", "ExecMainStartTimestamp": "Thu 2020-10-15 16:32:47 UTC", "ExecMainStartTimestampMonotonic": "17810924", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-10-15 16:32:47 UTC] ; stop_time=[n/a] ; pid=1347 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-10-15 16:32:47 UTC", "InactiveExitTimestampMonotonic": "17810966", "InvocationID": "6489d4952c5848edafde08dfef3c02dc", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-10-15T16:34:16.269 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1347", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-10-15T16:34:16.271 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-10-15 16:32:47 UTC", "StateChangeTimestampMonotonic": "17810966", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-10-15 16:32:47 UTC", "WatchdogTimestampMonotonic": "17810965", "WatchdogUSec": "0"}} 2020-10-15T16:34:16.301 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.241) 0:01:16.260 ****** 2020-10-15T16:34:16.308 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-10-15T16:34:16.310 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-10-15T16:34:16.344 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.042) 0:01:16.303 ****** 2020-10-15T16:34:16.351 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-10-15T16:34:16.353 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.387 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.042) 0:01:16.345 ****** 2020-10-15T16:34:16.415 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-10-15T16:34:16.417 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.453 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.066) 0:01:16.412 ****** 2020-10-15T16:34:16.481 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-10-15T16:34:16.482 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.514 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.061) 0:01:16.473 ****** 2020-10-15T16:34:16.541 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.575 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.060) 0:01:16.534 ****** 2020-10-15T16:34:16.602 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-10-15T16:34:16.604 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.641 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.065) 0:01:16.600 ****** 2020-10-15T16:34:16.668 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-10-15T16:34:16.670 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.702 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.060) 0:01:16.660 ****** 2020-10-15T16:34:16.728 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-10-15T16:34:16.730 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.777 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.075) 0:01:16.735 ****** 2020-10-15T16:34:16.805 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-10-15T16:34:16.807 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.839 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.062) 0:01:16.798 ****** 2020-10-15T16:34:16.867 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-10-15T16:34:16.869 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.902 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.062) 0:01:16.860 ****** 2020-10-15T16:34:16.929 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-10-15T16:34:16.931 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:16.963 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:16 +0000 (0:00:00.061) 0:01:16.922 ****** 2020-10-15T16:34:16.993 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-10-15T16:34:16.994 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-10-15T16:34:17.027 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.064) 0:01:16.986 ****** 2020-10-15T16:34:17.065 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-10-15T16:34:17.067 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-10-15T16:34:17.101 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.073) 0:01:17.059 ****** 2020-10-15T16:34:17.139 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-10-15T16:34:17.141 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-10-15T16:34:17.173 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.072) 0:01:17.132 ****** 2020-10-15T16:34:17.222 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-10-15T16:34:17.223 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-10-15T16:34:17.259 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.085) 0:01:17.217 ****** 2020-10-15T16:34:17.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-10-15T16:34:17.267 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:34:17.302 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.043) 0:01:17.261 ****** 2020-10-15T16:34:17.330 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-10-15T16:34:17.362 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.059) 0:01:17.321 ****** 2020-10-15T16:34:17.389 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-10-15T16:34:17.391 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-10-15T16:34:17.423 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.060) 0:01:17.382 ****** 2020-10-15T16:34:17.524 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-10-15T16:34:17.526 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.027596", "end": "2020-10-15 16:34:17.663506", "rc": 0, "start": "2020-10-15 16:34:17.635910", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:17.560 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.137) 0:01:17.519 ****** 2020-10-15T16:34:17.567 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-10-15T16:34:17.569 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "name": "ubuntu", "state": "present", "system": false} 2020-10-15T16:34:17.604 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.043) 0:01:17.562 ****** 2020-10-15T16:34:17.632 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-10-15T16:34:17.634 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:17.668 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.063) 0:01:17.626 ****** 2020-10-15T16:34:17.696 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-10-15T16:34:17.698 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:17.731 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.062) 0:01:17.689 ****** 2020-10-15T16:34:17.883 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-10-15T16:34:17.885 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-10-15T16:34:17.917 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:17 +0000 (0:00:00.186) 0:01:17.876 ****** 2020-10-15T16:34:18.291 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-10-15T16:34:18.293 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-10-15T16:34:18.326 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:18 +0000 (0:00:00.408) 0:01:18.285 ****** 2020-10-15T16:34:18.354 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-10-15T16:34:18.356 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi038"}, "changed": false} 2020-10-15T16:34:18.388 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:18 +0000 (0:00:00.061) 0:01:18.346 ****** 2020-10-15T16:34:18.851 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-10-15T16:34:18.853 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi038.front.sepia.ceph.com", "ansible_hostname": "smithi038", "ansible_nodename": "smithi038"}, "changed": true, "name": "smithi038"} 2020-10-15T16:34:18.886 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:18 +0000 (0:00:00.498) 0:01:18.844 ****** 2020-10-15T16:34:19.216 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-10-15T16:34:19.217 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-10-15T16:34:19.250 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:19 +0000 (0:00:00.364) 0:01:19.209 ****** 2020-10-15T16:34:19.558 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-10-15T16:34:19.559 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-10-15T16:34:19.591 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:19 +0000 (0:00:00.340) 0:01:19.550 ****** 2020-10-15T16:34:20.097 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-10-15T16:34:20.099 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-10-15T16:34:20.131 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.539) 0:01:20.089 ****** 2020-10-15T16:34:20.157 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-10-15T16:34:20.159 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.190 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.059) 0:01:20.149 ****** 2020-10-15T16:34:20.217 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-15T16:34:20.219 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.251 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.209 ****** 2020-10-15T16:34:20.278 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-15T16:34:20.280 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.311 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.270 ****** 2020-10-15T16:34:20.338 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-10-15T16:34:20.340 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.372 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.061) 0:01:20.331 ****** 2020-10-15T16:34:20.399 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-10-15T16:34:20.401 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.435 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.062) 0:01:20.393 ****** 2020-10-15T16:34:20.462 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-15T16:34:20.464 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.495 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.454 ****** 2020-10-15T16:34:20.522 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-15T16:34:20.524 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.556 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.514 ****** 2020-10-15T16:34:20.583 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-10-15T16:34:20.585 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.616 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.575 ****** 2020-10-15T16:34:20.643 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-10-15T16:34:20.677 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.635 ****** 2020-10-15T16:34:20.704 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-10-15T16:34:20.705 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-10-15T16:34:20.738 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.696 ****** 2020-10-15T16:34:20.769 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.031) 0:01:20.727 ****** 2020-10-15T16:34:20.776 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-10-15T16:34:20.810 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.041) 0:01:20.769 ****** 2020-10-15T16:34:20.838 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-10-15T16:34:20.839 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.871 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.060) 0:01:20.830 ****** 2020-10-15T16:34:20.908 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi038.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.910 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:20.943 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.071) 0:01:20.901 ****** 2020-10-15T16:34:20.971 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-10-15T16:34:20.972 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.005 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:20 +0000 (0:00:00.061) 0:01:20.963 ****** 2020-10-15T16:34:21.032 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:21.033 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.065 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.060) 0:01:21.024 ****** 2020-10-15T16:34:21.092 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-10-15T16:34:21.094 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.125 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.059) 0:01:21.084 ****** 2020-10-15T16:34:21.162 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-10-15T16:34:21.164 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.199 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.074) 0:01:21.158 ****** 2020-10-15T16:34:21.226 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-10-15T16:34:21.228 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.260 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.060) 0:01:21.218 ****** 2020-10-15T16:34:21.287 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-10-15T16:34:21.289 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.323 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.063) 0:01:21.282 ****** 2020-10-15T16:34:21.350 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-15T16:34:21.352 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.386 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.062) 0:01:21.344 ****** 2020-10-15T16:34:21.412 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-15T16:34:21.414 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.450 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.064) 0:01:21.409 ****** 2020-10-15T16:34:21.479 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-15T16:34:21.481 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.513 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.062) 0:01:21.472 ****** 2020-10-15T16:34:21.542 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-10-15T16:34:21.543 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.576 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.062) 0:01:21.534 ****** 2020-10-15T16:34:21.605 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-15T16:34:21.606 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.639 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.063) 0:01:21.597 ****** 2020-10-15T16:34:21.668 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-15T16:34:21.669 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.702 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.063) 0:01:21.661 ****** 2020-10-15T16:34:21.729 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-10-15T16:34:21.731 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.762 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.060) 0:01:21.721 ****** 2020-10-15T16:34:21.789 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.821 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.059) 0:01:21.780 ****** 2020-10-15T16:34:21.848 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-10-15T16:34:21.849 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.880 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.058) 0:01:21.839 ****** 2020-10-15T16:34:21.907 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-10-15T16:34:21.909 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:21.943 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.062) 0:01:21.902 ****** 2020-10-15T16:34:21.970 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-10-15T16:34:21.971 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:22.003 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:21 +0000 (0:00:00.059) 0:01:21.961 ****** 2020-10-15T16:34:22.029 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-10-15T16:34:22.031 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:22.062 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.059) 0:01:22.021 ****** 2020-10-15T16:34:22.088 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-10-15T16:34:22.090 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:22.121 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.059) 0:01:22.080 ****** 2020-10-15T16:34:22.149 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:22.182 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.060) 0:01:22.141 ****** 2020-10-15T16:34:22.314 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-10-15T16:34:22.316 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003406", "end": "2020-10-15 16:34:22.037878", "rc": 0, "start": "2020-10-15 16:34:22.034472", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:22.348 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.165) 0:01:22.306 ****** 2020-10-15T16:34:22.374 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-10-15T16:34:22.408 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.059) 0:01:22.366 ****** 2020-10-15T16:34:22.735 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-10-15T16:34:22.737 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-10-15T16:34:22.769 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.361) 0:01:22.727 ****** 2020-10-15T16:34:22.796 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-10-15T16:34:22.798 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:22.829 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:22 +0000 (0:00:00.060) 0:01:22.787 ****** 2020-10-15T16:34:23.831 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-10-15T16:34:23.833 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-10-15T16:34:24.803 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-10-15T16:34:24.839 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:24 +0000 (0:00:02.010) 0:01:24.798 ****** 2020-10-15T16:34:25.771 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-10-15T16:34:25.772 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:25.806 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:25 +0000 (0:00:00.966) 0:01:25.764 ****** 2020-10-15T16:34:25.834 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-10-15T16:34:25.873 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:25 +0000 (0:00:00.066) 0:01:25.831 ****** 2020-10-15T16:34:26.036 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-10-15T16:34:26.038 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035627", "end": "2020-10-15 16:34:25.757600", "rc": 0, "start": "2020-10-15 16:34:25.721973", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:26.071 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:26 +0000 (0:00:00.198) 0:01:26.030 ****** 2020-10-15T16:34:29.058 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-10-15T16:34:29.060 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:29.092 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:29 +0000 (0:00:03.020) 0:01:29.050 ****** 2020-10-15T16:34:29.328 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-10-15T16:34:29.330 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137591 files and directorie 2020-10-15T16:34:29.332 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-10-15T16:34:29.333 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137591 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-15T16:34:29.363 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:29 +0000 (0:00:00.271) 0:01:29.321 ****** 2020-10-15T16:34:30.227 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-15T16:34:30.229 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:34:30.260 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:30 +0000 (0:00:00.897) 0:01:30.219 ****** 2020-10-15T16:34:30.287 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-10-15T16:34:30.289 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:30.321 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:30 +0000 (0:00:00.060) 0:01:30.280 ****** 2020-10-15T16:34:38.247 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-10-15T16:34:38.250 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137591 files and directorie 2020-10-15T16:34:38.251 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-10-15T16:34:38.253 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137591 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-15T16:34:38.282 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:38 +0000 (0:00:07.960) 0:01:38.240 ****** 2020-10-15T16:34:38.754 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-15T16:34:38.755 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 18 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]\nFetched 10.4 MB in 1s (9566 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-10-15T16:34:38.756 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137449 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-10-15T16:34:38.758 INFO:teuthology.task.ansible.out:u1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 18 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]", "Fetched 10.4 MB in 1s (9566 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-10-15T16:34:38.759 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137449 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-15T16:34:38.790 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:38 +0000 (0:00:00.507) 0:01:38.748 ****** 2020-10-15T16:34:39.199 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-15T16:34:39.201 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:39.239 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:39 +0000 (0:00:00.449) 0:01:39.197 ****** 2020-10-15T16:34:39.267 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-10-15T16:34:39.268 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:39.300 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:39 +0000 (0:00:00.060) 0:01:39.258 ****** 2020-10-15T16:34:39.760 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-10-15T16:34:39.762 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:34:39.794 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:39 +0000 (0:00:00.494) 0:01:39.753 ****** 2020-10-15T16:34:39.904 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-10-15T16:34:39.905 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-10-15T16:34:39.937 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:39 +0000 (0:00:00.142) 0:01:39.896 ****** 2020-10-15T16:34:39.964 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-10-15T16:34:39.965 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:39.997 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:39 +0000 (0:00:00.059) 0:01:39.955 ****** 2020-10-15T16:34:40.116 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-10-15T16:34:40.118 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-10-15T16:34:40.149 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:40 +0000 (0:00:00.152) 0:01:40.107 ****** 2020-10-15T16:34:40.464 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-10-15T16:34:40.465 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-10-15T16:34:40.496 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:40 +0000 (0:00:00.347) 0:01:40.455 ****** 2020-10-15T16:34:40.648 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-10-15T16:34:40.650 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1230, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-10-15T16:34:40.681 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:40 +0000 (0:00:00.184) 0:01:40.640 ****** 2020-10-15T16:34:40.823 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-10-15T16:34:40.825 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:34:40.856 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:40 +0000 (0:00:00.175) 0:01:40.815 ****** 2020-10-15T16:34:41.131 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-10-15T16:34:41.132 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-10-15 16:32:52 UTC", "ActiveEnterTimestampMonotonic": "20991581", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target network.target system.slice remote-fs.target systemd-journald.socket nss-lookup.target sysinit.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-10-15 16:32:52 UTC", "AssertTimestampMonotonic": "20403756", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-10-15 16:32:52 UTC", "ConditionTimestampMonotonic": "20403756", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-10-15T16:34:41.134 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1394", "ExecMainStartTimestamp": "Thu 2020-10-15 16:32:52 UTC", "ExecMainStartTimestampMonotonic": "20991567", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-10-15 16:32:52 UTC", "InactiveExitTimestampMonotonic": "20404487", "InvocationID": "d844e4c3729540fcb2552e4cfdd9248d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-10-15T16:34:41.135 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1394", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-10-15T16:34:41.137 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-10-15 16:32:52 UTC", "StateChangeTimestampMonotonic": "20991581", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-10-15 16:32:52 UTC", "WatchdogTimestampMonotonic": "20991579", "WatchdogUSec": "0"}} 2020-10-15T16:34:41.166 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.309) 0:01:41.125 ****** 2020-10-15T16:34:41.193 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-15T16:34:41.195 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.226 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.059) 0:01:41.185 ****** 2020-10-15T16:34:41.253 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.289 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.062) 0:01:41.247 ****** 2020-10-15T16:34:41.317 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.349 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.060) 0:01:41.308 ****** 2020-10-15T16:34:41.377 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-10-15T16:34:41.378 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.410 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.060) 0:01:41.368 ****** 2020-10-15T16:34:41.437 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-15T16:34:41.438 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.470 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.059) 0:01:41.428 ****** 2020-10-15T16:34:41.496 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-15T16:34:41.498 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.529 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.059) 0:01:41.488 ****** 2020-10-15T16:34:41.556 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-15T16:34:41.558 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.592 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.062) 0:01:41.550 ****** 2020-10-15T16:34:41.618 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-15T16:34:41.620 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.651 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.059) 0:01:41.610 ****** 2020-10-15T16:34:41.678 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:41.680 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.712 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.060) 0:01:41.670 ****** 2020-10-15T16:34:41.738 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-15T16:34:41.740 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.772 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.059) 0:01:41.730 ****** 2020-10-15T16:34:41.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-15T16:34:41.800 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.831 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.059) 0:01:41.790 ****** 2020-10-15T16:34:41.858 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-15T16:34:41.859 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.891 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.060) 0:01:41.850 ****** 2020-10-15T16:34:41.918 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:41.952 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:41 +0000 (0:00:00.060) 0:01:41.910 ****** 2020-10-15T16:34:41.978 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-15T16:34:41.980 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.011 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.060) 0:01:41.970 ****** 2020-10-15T16:34:42.038 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-15T16:34:42.040 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.071 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.059) 0:01:42.030 ****** 2020-10-15T16:34:42.098 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-15T16:34:42.100 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.136 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.064) 0:01:42.094 ****** 2020-10-15T16:34:42.166 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-10-15T16:34:42.168 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.209 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.073) 0:01:42.167 ****** 2020-10-15T16:34:42.236 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-15T16:34:42.238 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.271 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.062) 0:01:42.229 ****** 2020-10-15T16:34:42.298 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-15T16:34:42.300 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.335 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.064) 0:01:42.294 ****** 2020-10-15T16:34:42.363 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-15T16:34:42.365 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.399 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.063) 0:01:42.357 ****** 2020-10-15T16:34:42.426 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-15T16:34:42.428 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.460 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.061) 0:01:42.419 ****** 2020-10-15T16:34:42.487 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-15T16:34:42.488 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.520 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.060) 0:01:42.479 ****** 2020-10-15T16:34:42.546 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-15T16:34:42.548 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.580 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.059) 0:01:42.538 ****** 2020-10-15T16:34:42.606 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-15T16:34:42.608 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.639 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.059) 0:01:42.598 ****** 2020-10-15T16:34:42.666 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-15T16:34:42.667 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.699 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.059) 0:01:42.657 ****** 2020-10-15T16:34:42.726 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-10-15T16:34:42.727 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:42.759 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.060) 0:01:42.718 ****** 2020-10-15T16:34:42.893 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-10-15T16:34:42.895 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-10-15T16:34:42.927 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:42 +0000 (0:00:00.167) 0:01:42.886 ****** 2020-10-15T16:34:43.234 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-10-15T16:34:43.235 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-10-15T16:34:43.267 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:43 +0000 (0:00:00.340) 0:01:43.226 ****** 2020-10-15T16:34:43.294 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-15T16:34:43.296 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:43.327 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:43 +0000 (0:00:00.059) 0:01:43.286 ****** 2020-10-15T16:34:43.634 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-10-15T16:34:43.636 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-10-15T16:34:43.668 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:43 +0000 (0:00:00.340) 0:01:43.626 ****** 2020-10-15T16:34:43.964 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-10-15T16:34:43.966 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-10-15T16:34:43.998 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:43 +0000 (0:00:00.329) 0:01:43.956 ****** 2020-10-15T16:34:44.294 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-10-15T16:34:44.296 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-10-15T16:34:44.328 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.330) 0:01:44.287 ****** 2020-10-15T16:34:44.458 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-10-15T16:34:44.460 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:34:44.499 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.171) 0:01:44.458 ****** 2020-10-15T16:34:44.528 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.565 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.065) 0:01:44.523 ****** 2020-10-15T16:34:44.592 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-10-15T16:34:44.593 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.625 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.059) 0:01:44.583 ****** 2020-10-15T16:34:44.650 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-10-15T16:34:44.652 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.683 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.058) 0:01:44.642 ****** 2020-10-15T16:34:44.709 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-10-15T16:34:44.710 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.742 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.058) 0:01:44.700 ****** 2020-10-15T16:34:44.768 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.801 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.059) 0:01:44.760 ****** 2020-10-15T16:34:44.827 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:44.829 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-10-15T16:34:44.864 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:44 +0000 (0:00:00.063) 0:01:44.823 ****** 2020-10-15T16:34:44.901 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-10-15T16:34:44.903 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.912 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.913 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.924 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.935 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.936 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.946 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-10-15T16:34:44.956 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.958 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.967 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.969 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.978 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.989 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:44.990 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:45.000 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:45.037 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:45 +0000 (0:00:00.172) 0:01:44.995 ****** 2020-10-15T16:34:45.219 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-10-15T16:34:45.221 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003227", "end": "2020-10-15 16:34:45.195460", "rc": 0, "start": "2020-10-15 16:34:45.192233", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-10-15T16:34:45.257 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:45 +0000 (0:00:00.220) 0:01:45.216 ****** 2020-10-15T16:34:45.284 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-10-15T16:34:45.286 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:45.325 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:45 +0000 (0:00:00.067) 0:01:45.283 ****** 2020-10-15T16:34:45.352 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-10-15T16:34:45.353 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:45.386 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:45 +0000 (0:00:00.060) 0:01:45.344 ****** 2020-10-15T16:34:45.412 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-10-15T16:34:45.414 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:45.446 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:45 +0000 (0:00:00.060) 0:01:45.405 ****** 2020-10-15T16:34:46.367 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-10-15T16:34:46.369 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:34:46.405 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:46 +0000 (0:00:00.958) 0:01:46.363 ****** 2020-10-15T16:34:46.431 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-10-15T16:34:46.433 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:46.466 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:46 +0000 (0:00:00.060) 0:01:46.424 ****** 2020-10-15T16:34:46.503 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-15T16:34:46.505 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item={'block_used': 13749362, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226345746, 'size_available': 927112175616, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149457, 'block_size': 4096, 'inode_available': 60905519}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226345746, "block_size": 4096, "block_total": 240095108, "block_used": 13749362, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905519, "inode_total": 61054976, "inode_used": 149457, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927112175616, "size_total": 983429562368, "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}} 2020-10-15T16:34:46.558 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:46 +0000 (0:00:00.092) 0:01:46.517 ****** 2020-10-15T16:34:46.702 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-10-15T16:34:46.704 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010031", "end": "2020-10-15 16:34:46.826038", "rc": 0, "start": "2020-10-15 16:34:46.816007", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:46.736 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:46 +0000 (0:00:00.177) 0:01:46.694 ****** 2020-10-15T16:34:46.774 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-10-15T16:34:46.775 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'block_used': 13749362, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226345746, 'size_available': 927112175616, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149457, 'block_size': 4096, 'inode_available': 60905519}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226345746, "block_size": 4096, "block_total": 240095108, "block_used": 13749362, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905519, "inode_total": 61054976, "inode_used": 149457, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927112175616, "size_total": 983429562368, "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:46.807 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:46 +0000 (0:00:00.071) 0:01:46.766 ****** 2020-10-15T16:34:46.834 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-10-15T16:34:46.872 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:46 +0000 (0:00:00.064) 0:01:46.831 ****** 2020-10-15T16:34:46.911 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:46.913 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:48.527 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-15T16:34:48.529 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 18 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]\nFetched 10.4 MB in 1s (8888 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-10-15T16:34:48.530 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137449 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-10-15T16:34:48.532 INFO:teuthology.task.ansible.out:u1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 18 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]", "Fetched 10.4 MB in 1s (8888 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-10-15T16:34:48.533 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137449 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-15T16:34:48.563 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:48 +0000 (0:00:01.690) 0:01:48.521 ****** 2020-10-15T16:34:48.914 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:48.916 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5trioyDzi2GhLIB06aDQAhGWcLycHvaVbh'], 'uuids': ['41eb3482-66de-494f-9920-b3e68760e6f7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.878683", "end": "2020-10-15 16:34:49.034420", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5trioyDzi2GhLIB06aDQAhGWcLycHvaVbh"], "labels": [], "masters": [], "uuids": ["41eb3482-66de-494f-9920-b3e68760e6f7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:47.155737", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:34:48.926 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:48.927 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:49.496 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-10-15T16:34:49.497 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:49.532 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:49 +0000 (0:00:00.969) 0:01:49.491 ****** 2020-10-15T16:34:49.653 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-10-15T16:34:49.654 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-10-15T16:34:49.685 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:49 +0000 (0:00:00.153) 0:01:49.644 ****** 2020-10-15T16:34:49.714 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:49.747 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:49 +0000 (0:00:00.061) 0:01:49.705 ****** 2020-10-15T16:34:49.878 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-10-15T16:34:49.911 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:49 +0000 (0:00:00.163) 0:01:49.869 ****** 2020-10-15T16:34:50.175 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:50.177 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5tGbcws1LgGtP2AY9BDY3yxgS8fHaAT65D'], 'uuids': ['c5e3908f-b0a3-47da-ab2d-cf2344b0b7a0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.127344", "end": "2020-10-15 16:34:50.295198", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5tGbcws1LgGtP2AY9BDY3yxgS8fHaAT65D"], "labels": [], "masters": [], "uuids": ["c5e3908f-b0a3-47da-ab2d-cf2344b0b7a0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:49.167854", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:34:50.177 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => 2020-10-15T16:34:50.179 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFFN', 'wwn-0x5000c50092929219'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFFN-part1', 'wwn-0x5000c50092929219-part1'], 'uuids': ['6a302439-42e1-4f94-b095-dec7b770edc2']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092929219', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFFN', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFFN", "wwn-0x5000c50092929219"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFFN-part1", "wwn-0x5000c50092929219-part1"], "labels": [], "masters": [], "uuids": ["6a302439-42e1-4f94-b095-dec7b770edc2"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFFN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092929219"}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:50.221 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-10-15T16:34:50.222 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-10-15T16:34:50.254 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:50 +0000 (0:00:00.343) 0:01:50.212 ****** 2020-10-15T16:34:50.405 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-10-15T16:34:50.407 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1230, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-10-15T16:34:50.439 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:50 +0000 (0:00:00.185) 0:01:50.397 ****** 2020-10-15T16:34:50.570 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:34:50.602 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:50 +0000 (0:00:00.163) 0:01:50.561 ****** 2020-10-15T16:34:50.899 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-10-15T16:34:50.900 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-10-15 16:32:48 UTC", "ActiveEnterTimestampMonotonic": "18361525", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service nss-lookup.target sysinit.target network.target basic.target systemd-journald.socket remote-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-10-15 16:32:47 UTC", "AssertTimestampMonotonic": "17811132", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-10-15 16:32:47 UTC", "ConditionTimestampMonotonic": "17811132", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-10-15T16:34:50.902 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1458", "ExecMainStartTimestamp": "Thu 2020-10-15 16:32:48 UTC", "ExecMainStartTimestampMonotonic": "18361514", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-10-15 16:32:47 UTC", "InactiveExitTimestampMonotonic": "17812163", "InvocationID": "d0583f08339c498cbf5b4f5d9ca6b5f5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-10-15T16:34:50.905 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1458", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-10-15T16:34:50.906 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-10-15 16:32:48 UTC", "StateChangeTimestampMonotonic": "18361525", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-10-15 16:32:48 UTC", "WatchdogTimestampMonotonic": "18361524", "WatchdogUSec": "0"}} 2020-10-15T16:34:50.942 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:50 +0000 (0:00:00.339) 0:01:50.901 ****** 2020-10-15T16:34:50.973 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-15T16:34:50.974 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.008 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.065) 0:01:50.967 ****** 2020-10-15T16:34:51.036 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-15T16:34:51.037 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.072 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.063) 0:01:51.030 ****** 2020-10-15T16:34:51.099 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.135 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.062) 0:01:51.093 ****** 2020-10-15T16:34:51.162 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-10-15T16:34:51.163 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.196 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.061) 0:01:51.154 ****** 2020-10-15T16:34:51.224 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-15T16:34:51.226 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.259 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.062) 0:01:51.217 ****** 2020-10-15T16:34:51.287 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-15T16:34:51.288 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.321 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.062) 0:01:51.280 ****** 2020-10-15T16:34:51.350 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-15T16:34:51.352 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.384 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.062) 0:01:51.342 ****** 2020-10-15T16:34:51.411 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-15T16:34:51.413 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.445 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.061) 0:01:51.404 ****** 2020-10-15T16:34:51.463 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:51.464 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-hWaQJa-zzYe-e5Ay-mVgh-ktMO-Iw8W-7bSLuJ', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN', 'nvme-nvme.8086-50484654363230353030514a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.167294", "end": "2020-10-15 16:34:51.572902", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-hWaQJa-zzYe-e5Ay-mVgh-ktMO-Iw8W-7bSLuJ", "nvme-INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN", "nvme-nvme.8086-50484654363230353030514a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:50.405608", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-10-15T16:34:51.466 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.477 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.513 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.068) 0:01:51.472 ****** 2020-10-15T16:34:51.521 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:51.523 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.533 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.568 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.054) 0:01:51.527 ****** 2020-10-15T16:34:51.595 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-15T16:34:51.597 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.631 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.062) 0:01:51.589 ****** 2020-10-15T16:34:51.658 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-15T16:34:51.659 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.691 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.060) 0:01:51.650 ****** 2020-10-15T16:34:51.718 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-15T16:34:51.720 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.751 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.060) 0:01:51.710 ****** 2020-10-15T16:34:51.779 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-15T16:34:51.780 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.816 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.064) 0:01:51.775 ****** 2020-10-15T16:34:51.843 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-15T16:34:51.845 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.878 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.061) 0:01:51.836 ****** 2020-10-15T16:34:51.905 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-15T16:34:51.907 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:51.940 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.062) 0:01:51.899 ****** 2020-10-15T16:34:51.967 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-10-15T16:34:51.969 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.002 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:51 +0000 (0:00:00.061) 0:01:51.960 ****** 2020-10-15T16:34:52.029 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-15T16:34:52.031 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.064 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.061) 0:01:52.022 ****** 2020-10-15T16:34:52.093 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-15T16:34:52.095 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.127 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.062) 0:01:52.085 ****** 2020-10-15T16:34:52.154 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-15T16:34:52.156 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.187 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.060) 0:01:52.145 ****** 2020-10-15T16:34:52.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-15T16:34:52.216 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.248 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.060) 0:01:52.206 ****** 2020-10-15T16:34:52.275 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-15T16:34:52.276 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.308 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.060) 0:01:52.267 ****** 2020-10-15T16:34:52.335 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-15T16:34:52.337 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.368 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.060) 0:01:52.327 ****** 2020-10-15T16:34:52.396 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-15T16:34:52.397 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.429 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.060) 0:01:52.387 ****** 2020-10-15T16:34:52.456 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-15T16:34:52.458 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.497 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.068) 0:01:52.455 ****** 2020-10-15T16:34:52.525 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-10-15T16:34:52.527 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.564 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.067) 0:01:52.523 ****** 2020-10-15T16:34:52.698 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-10-15T16:34:52.700 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-10-15T16:34:52.733 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:52 +0000 (0:00:00.168) 0:01:52.691 ****** 2020-10-15T16:34:52.774 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:52.776 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5toHhi7odNHHnx4ouWCEO0mXhA5wt7pKXM'], 'uuids': ['79a021ce-3d94-44bd-a598-b5fe1c8aa258']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.166117", "end": "2020-10-15 16:34:52.887402", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5toHhi7odNHHnx4ouWCEO0mXhA5wt7pKXM"], "labels": [], "masters": [], "uuids": ["79a021ce-3d94-44bd-a598-b5fe1c8aa258"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:51.721285", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:34:52.788 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:52.799 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:53.052 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-10-15T16:34:53.054 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-10-15T16:34:53.086 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:53 +0000 (0:00:00.352) 0:01:53.044 ****** 2020-10-15T16:34:53.113 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-15T16:34:53.115 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:53.147 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:53 +0000 (0:00:00.061) 0:01:53.105 ****** 2020-10-15T16:34:53.485 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-10-15T16:34:53.487 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-10-15T16:34:53.519 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:53 +0000 (0:00:00.372) 0:01:53.478 ****** 2020-10-15T16:34:53.835 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-10-15T16:34:53.837 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-10-15T16:34:53.868 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:53 +0000 (0:00:00.349) 0:01:53.827 ****** 2020-10-15T16:34:54.071 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:54.073 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t33IQh1rnoJqq9SuZ0Yu1XgD0HqVWp0OT'], 'uuids': ['73266a59-542f-4b84-bab9-0cd29b6a670f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.156659", "end": "2020-10-15 16:34:54.186214", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t33IQh1rnoJqq9SuZ0Yu1XgD0HqVWp0OT"], "labels": [], "masters": [], "uuids": ["73266a59-542f-4b84-bab9-0cd29b6a670f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:53.029555", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi183.front.sepia.ceph.com] => 2020-10-15T16:34:54.075 INFO:teuthology.task.ansible.out:(item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.177 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-10-15T16:34:54.179 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-10-15T16:34:54.212 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.344) 0:01:54.171 ****** 2020-10-15T16:34:54.344 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-10-15T16:34:54.346 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:34:54.378 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.165) 0:01:54.336 ****** 2020-10-15T16:34:54.405 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-10-15T16:34:54.407 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.442 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.063) 0:01:54.400 ****** 2020-10-15T16:34:54.469 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-10-15T16:34:54.471 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.510 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.068) 0:01:54.469 ****** 2020-10-15T16:34:54.538 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-10-15T16:34:54.540 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.572 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.061) 0:01:54.531 ****** 2020-10-15T16:34:54.600 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-10-15T16:34:54.601 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.634 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.061) 0:01:54.592 ****** 2020-10-15T16:34:54.662 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-10-15T16:34:54.663 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.705 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.070) 0:01:54.663 ****** 2020-10-15T16:34:54.733 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:54.734 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-10-15T16:34:54.768 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.063) 0:01:54.727 ****** 2020-10-15T16:34:54.806 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-10-15T16:34:54.808 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.817 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.818 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.828 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.829 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.839 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.850 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-10-15T16:34:54.860 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.862 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.871 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.882 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.884 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.893 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.895 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.904 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-10-15T16:34:54.943 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:54 +0000 (0:00:00.174) 0:01:54.902 ****** 2020-10-15T16:34:55.149 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-10-15T16:34:55.150 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003675", "end": "2020-10-15 16:34:55.121574", "rc": 0, "start": "2020-10-15 16:34:55.117899", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-10-15T16:34:55.185 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:55 +0000 (0:00:00.241) 0:01:55.144 ****** 2020-10-15T16:34:55.213 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-10-15T16:34:55.214 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:55.250 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:55 +0000 (0:00:00.064) 0:01:55.208 ****** 2020-10-15T16:34:55.277 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-10-15T16:34:55.279 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:55.311 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:55 +0000 (0:00:00.061) 0:01:55.270 ****** 2020-10-15T16:34:55.330 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:55.331 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t0PD9KsbGJkqQoPoIOh1Lj233eGS1tIeD'], 'uuids': ['df38c418-2a95-4129-8ff8-b51c4d87ceed']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.122914", "end": "2020-10-15 16:34:55.436279", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t0PD9KsbGJkqQoPoIOh1Lj233eGS1tIeD"], "labels": [], "masters": [], "uuids": ["df38c418-2a95-4129-8ff8-b51c4d87ceed"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:54.313365", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:34:55.340 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-10-15T16:34:55.342 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:55.374 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:55 +0000 (0:00:00.062) 0:01:55.332 ****** 2020-10-15T16:34:55.407 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:55 +0000 (0:00:00.032) 0:01:55.365 ****** 2020-10-15T16:34:55.445 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:34:55.447 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:55.717 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5trioyDzi2GhLIB06aDQAhGWcLycHvaVbh'], 'uuids': ['41eb3482-66de-494f-9920-b3e68760e6f7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.151308", "end": "2020-10-15 16:34:55.832920", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5trioyDzi2GhLIB06aDQAhGWcLycHvaVbh"], "labels": [], "masters": [], "uuids": ["41eb3482-66de-494f-9920-b3e68760e6f7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:55.681612", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14663 s, 787 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14663 s, 787 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:55.728 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:55.741 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.022 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5tGbcws1LgGtP2AY9BDY3yxgS8fHaAT65D'], 'uuids': ['c5e3908f-b0a3-47da-ab2d-cf2344b0b7a0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.159057", "end": "2020-10-15 16:34:56.130349", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5tGbcws1LgGtP2AY9BDY3yxgS8fHaAT65D"], "labels": [], "masters": [], "uuids": ["c5e3908f-b0a3-47da-ab2d-cf2344b0b7a0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:55.971292", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154333 s, 747 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154333 s, 747 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:56.023 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFFN', 'wwn-0x5000c50092929219'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFFN-part1', 'wwn-0x5000c50092929219-part1'], 'uuids': ['6a302439-42e1-4f94-b095-dec7b770edc2']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092929219', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFFN', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFFN", "wwn-0x5000c50092929219"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFFN-part1", "wwn-0x5000c50092929219-part1"], "labels": [], "masters": [], "uuids": ["6a302439-42e1-4f94-b095-dec7b770edc2"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFFN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092929219"}}, "skip_reason": 2020-10-15T16:34:56.025 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-15T16:34:56.201 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-hWaQJa-zzYe-e5Ay-mVgh-ktMO-Iw8W-7bSLuJ', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN', 'nvme-nvme.8086-50484654363230353030514a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049641", "end": "2020-10-15 16:34:56.312539", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-hWaQJa-zzYe-e5Ay-mVgh-ktMO-Iw8W-7bSLuJ", "nvme-INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN", "nvme-nvme.8086-50484654363230353030514a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:56.262898", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0450658 s, 2.6 GB/s", "std 2020-10-15T16:34:56.203 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0450658 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.214 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.287 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-10-15T16:34:56.289 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:34:56.325 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:56 +0000 (0:00:00.917) 0:01:56.283 ****** 2020-10-15T16:34:56.352 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-10-15T16:34:56.354 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.387 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:56 +0000 (0:00:00.062) 0:01:56.345 ****** 2020-10-15T16:34:56.425 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-15T16:34:56.426 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item={'block_used': 13749371, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226345737, 'size_available': 927112138752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149456, 'block_size': 4096, 'inode_available': 60905520}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226345737, "block_size": 4096, "block_total": 240095108, "block_used": 13749371, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905520, "inode_total": 61054976, "inode_used": 149456, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927112138752, "size_total": 983429562368, "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}} 2020-10-15T16:34:56.460 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:56 +0000 (0:00:00.073) 0:01:56.419 ****** 2020-10-15T16:34:56.489 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:34:56.491 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5toHhi7odNHHnx4ouWCEO0mXhA5wt7pKXM'], 'uuids': ['79a021ce-3d94-44bd-a598-b5fe1c8aa258']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.153044", "end": "2020-10-15 16:34:56.597409", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5toHhi7odNHHnx4ouWCEO0mXhA5wt7pKXM"], "labels": [], "masters": [], "uuids": ["79a021ce-3d94-44bd-a598-b5fe1c8aa258"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:56.444365", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148727 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148727 s, 776 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': ' 2020-10-15T16:34:56.492 INFO:teuthology.task.ansible.out:0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.501 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.595 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-10-15T16:34:56.597 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009876", "end": "2020-10-15 16:34:56.335064", "rc": 0, "start": "2020-10-15 16:34:56.325188", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:56.630 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:56 +0000 (0:00:00.169) 0:01:56.588 ****** 2020-10-15T16:34:56.667 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-10-15T16:34:56.669 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'block_used': 13749371, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226345737, 'size_available': 927112138752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149456, 'block_size': 4096, 'inode_available': 60905520}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226345737, "block_size": 4096, "block_total": 240095108, "block_used": 13749371, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905520, "inode_total": 61054976, "inode_used": 149456, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927112138752, "size_total": 983429562368, "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.701 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:56 +0000 (0:00:00.071) 0:01:56.660 ****** 2020-10-15T16:34:56.728 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-10-15T16:34:56.762 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:56 +0000 (0:00:00.060) 0:01:56.720 ****** 2020-10-15T16:34:56.769 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:34:56.770 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t33IQh1rnoJqq9SuZ0Yu1XgD0HqVWp0OT'], 'uuids': ['73266a59-542f-4b84-bab9-0cd29b6a670f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.151284", "end": "2020-10-15 16:34:56.880015", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t33IQh1rnoJqq9SuZ0Yu1XgD0HqVWp0OT"], "labels": [], "masters": [], "uuids": ["73266a59-542f-4b84-bab9-0cd29b6a670f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:56.728731", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148414 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148414 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:56.772 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:56.814 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:57.047 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:34:57.048 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t0PD9KsbGJkqQoPoIOh1Lj233eGS1tIeD'], 'uuids': ['df38c418-2a95-4129-8ff8-b51c4d87ceed']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.150061", "end": "2020-10-15 16:34:57.155614", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-bpjkgTxG1gcHilxn0ODuz8nC4PzP5J5t0PD9KsbGJkqQoPoIOh1Lj233eGS1tIeD"], "labels": [], "masters": [], "uuids": ["df38c418-2a95-4129-8ff8-b51c4d87ceed"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:57.005553", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147132 s, 784 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147132 s, 784 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:57.090 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.328) 0:01:57.049 ****** 2020-10-15T16:34:57.340 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-10-15T16:34:57.341 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.123959", "end": "2020-10-15 16:34:57.456708", "rc": 0, "start": "2020-10-15 16:34:57.332749", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:57.373 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.282) 0:01:57.331 ****** 2020-10-15T16:34:57.565 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-10-15T16:34:57.567 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.064849", "end": "2020-10-15 16:34:57.688539", "rc": 0, "start": "2020-10-15 16:34:57.623690", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:34:57.599 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.226) 0:01:57.557 ****** 2020-10-15T16:34:57.626 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-10-15T16:34:57.661 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.062) 0:01:57.620 ****** 2020-10-15T16:34:57.688 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-10-15T16:34:57.689 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:57.724 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.062) 0:01:57.682 ****** 2020-10-15T16:34:57.750 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-10-15T16:34:57.752 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:57.784 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.060) 0:01:57.742 ****** 2020-10-15T16:34:57.810 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-10-15T16:34:57.812 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:57.845 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.061) 0:01:57.804 ****** 2020-10-15T16:34:57.872 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-10-15T16:34:57.874 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:57.909 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:57 +0000 (0:00:00.063) 0:01:57.867 ****** 2020-10-15T16:34:58.030 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-10-15T16:34:58.031 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:34:58.064 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.155) 0:01:58.023 ****** 2020-10-15T16:34:58.102 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-15T16:34:58.104 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'block_used': 13749362, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226345746, 'size_available': 927112175616, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149457, 'block_size': 4096, 'inode_available': 60905519}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226345746, "block_size": 4096, "block_total": 240095108, "block_used": 13749362, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905519, "inode_total": 61054976, "inode_used": 149457, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927112175616, "size_total": 983429562368, "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.137 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.072) 0:01:58.095 ****** 2020-10-15T16:34:58.164 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-10-15T16:34:58.166 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.200 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.062) 0:01:58.158 ****** 2020-10-15T16:34:58.238 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:58.239 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.276 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.076) 0:01:58.235 ****** 2020-10-15T16:34:58.303 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:34:58.305 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.337 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.061) 0:01:58.296 ****** 2020-10-15T16:34:58.364 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-10-15T16:34:58.366 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.399 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.061) 0:01:58.357 ****** 2020-10-15T16:34:58.427 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-10-15T16:34:58.428 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.461 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.061) 0:01:58.419 ****** 2020-10-15T16:34:58.487 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-10-15T16:34:58.489 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.521 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:58 +0000 (0:00:00.060) 0:01:58.480 ****** 2020-10-15T16:34:58.923 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:34:58.925 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdZupzHZUu6WNiQyvAcbfv06vYa6pi8acR'], 'uuids': ['a951d209-77ba-4dc3-b487-4f92a7694431']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.985780", "end": "2020-10-15 16:34:58.648006", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdZupzHZUu6WNiQyvAcbfv06vYa6pi8acR"], "labels": [], "masters": [], "uuids": ["a951d209-77ba-4dc3-b487-4f92a7694431"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:56.662226", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi038.front.sepia.ceph.com] => 2020-10-15T16:34:58.927 INFO:teuthology.task.ansible.out:(item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:58.936 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:34:59.076 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-10-15T16:34:59.078 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-10-15T16:34:59.110 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:34:59 +0000 (0:00:00.588) 0:01:59.068 ****** 2020-10-15T16:34:59.644 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:34:59.646 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:34:59.863 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:00.069 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:00.276 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:35:00.277 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubddHfioysUgiTVKiZeJuJ04o74nvsp245a'], 'uuids': ['1f15b17b-5d26-487a-80d5-1e0375235256']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.204682", "end": "2020-10-15 16:34:59.997749", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubddHfioysUgiTVKiZeJuJ04o74nvsp245a"], "labels": [], "masters": [], "uuids": ["1f15b17b-5d26-487a-80d5-1e0375235256"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:34:58.793067", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:35:00.280 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D', 'wwn-0x5000c500871b5cd5'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D-part1', 'wwn-0x5000c500871b5cd5-part1'], 'uuids': ['6a302439-42e1-4f94-b095-dec7b770edc2']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b5cd5', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J33D', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D", "wwn-0x5000c500871b5cd5"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D-part1", "wwn-0x5000c500871b5cd5-part1"], "labels": [], "masters": [], "uuids": ["6a302439-42e1-4f94-b095-dec7b770edc2"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J33D", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b5cd5"}}, "skip_reason": 2020-10-15T16:35:00.281 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-15T16:35:00.292 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:35:00.293 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:00.489 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:00.528 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:00 +0000 (0:00:01.418) 0:02:00.487 ****** 2020-10-15T16:35:00.662 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-10-15T16:35:00.664 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-10-15T16:35:00.695 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:00 +0000 (0:00:00.167) 0:02:00.654 ****** 2020-10-15T16:35:00.847 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-15T16:35:00.849 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:00.976 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:01.091 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:01.206 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:01.217 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:01.253 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:01 +0000 (0:00:00.557) 0:02:01.211 ****** 2020-10-15T16:35:01.383 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-10-15T16:35:01.385 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-10-15T16:35:01.417 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:01 +0000 (0:00:00.164) 0:02:01.375 ****** 2020-10-15T16:35:01.454 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-10-15T16:35:01.455 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-10-15T16:35:01.489 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:01 +0000 (0:00:00.071) 0:02:01.447 ****** 2020-10-15T16:35:01.610 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:35:01.611 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-X9k38g-df55-vY91-KhVT-0Z2j-S0tH-rqmlv0', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.195501", "end": "2020-10-15 16:35:01.333943", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-X9k38g-df55-vY91-KhVT-0Z2j-S0tH-rqmlv0", "nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:00.138442", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT 2020-10-15T16:35:01.613 INFO:teuthology.task.ansible.out: data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:01.623 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:01.953 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-10-15T16:35:01.954 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true} 2020-10-15T16:35:01.987 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:01 +0000 (0:00:00.498) 0:02:01.945 ****** 2020-10-15T16:35:02.436 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-10-15T16:35:02.438 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-10-15T16:35:02.487 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:02 +0000 (0:00:00.500) 0:02:02.446 ****** 2020-10-15T16:35:02.644 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-10-15T16:35:02.645 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-10-15T16:35:02.691 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:02 +0000 (0:00:00.203) 0:02:02.649 ****** 2020-10-15T16:35:02.811 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-10-15T16:35:02.813 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-10-15T16:35:02.845 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:02 +0000 (0:00:00.154) 0:02:02.804 ****** 2020-10-15T16:35:02.893 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:35:02.895 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdpExDw5LWvsChoEPkj3wTPc8ZSFKkI3Un'], 'uuids': ['05528fd3-9a0e-467d-8715-130a109ccba3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.158175", "end": "2020-10-15 16:35:02.627895", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdpExDw5LWvsChoEPkj3wTPc8ZSFKkI3Un"], "labels": [], "masters": [], "uuids": ["05528fd3-9a0e-467d-8715-130a109ccba3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:01.469720", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:35:02.906 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:02.907 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:03.155 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-10-15T16:35:03.157 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-10-15T16:35:03.189 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:03 +0000 (0:00:00.343) 0:02:03.147 ****** 2020-10-15T16:35:03.496 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-10-15T16:35:03.497 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-10-15T16:35:03.529 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:03 +0000 (0:00:00.340) 0:02:03.488 ****** 2020-10-15T16:35:04.166 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:35:04.168 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4Cubddfkmifup8n8Y2yklK0MiAPDqHT7TMPLI'], 'uuids': ['619b5798-65c4-4cc2-ac34-2ef0cfa39202']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.135786", "end": "2020-10-15 16:35:03.900522", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4Cubddfkmifup8n8Y2yklK0MiAPDqHT7TMPLI"], "labels": [], "masters": [], "uuids": ["619b5798-65c4-4cc2-ac34-2ef0cfa39202"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:02.764736", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:35:04.178 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:04.427 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:35:04.460 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:04 +0000 (0:00:00.930) 0:02:04.418 ****** 2020-10-15T16:35:04.642 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-10-15T16:35:04.644 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047266", "end": "2020-10-15 16:35:04.763922", "rc": 0, "start": "2020-10-15 16:35:04.716656", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-10-15T16:35:04.676 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:04 +0000 (0:00:00.216) 0:02:04.634 ****** 2020-10-15T16:35:04.703 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-10-15T16:35:04.705 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:04.736 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:04 +0000 (0:00:00.059) 0:02:04.694 ****** 2020-10-15T16:35:04.763 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-10-15T16:35:04.765 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:04.795 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:04 +0000 (0:00:00.059) 0:02:04.754 ****** 2020-10-15T16:35:05.417 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-15T16:35:05.418 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdrMkvwWTV5sX2685bKoiVNG0vwDAjAKWB'], 'uuids': ['1db01386-704c-4ada-b277-492a1d9c8c17']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.119296", "end": "2020-10-15 16:35:05.151616", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdrMkvwWTV5sX2685bKoiVNG0vwDAjAKWB"], "labels": [], "masters": [], "uuids": ["1db01386-704c-4ada-b277-492a1d9c8c17"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:04.032320", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-15T16:35:05.462 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:05 +0000 (0:00:00.666) 0:02:05.421 ****** 2020-10-15T16:35:05.511 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:35:05.513 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:05.677 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-10-15T16:35:05.679 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:35:05.713 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:05 +0000 (0:00:00.250) 0:02:05.671 ****** 2020-10-15T16:35:05.739 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-10-15T16:35:05.741 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:05.773 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:05 +0000 (0:00:00.060) 0:02:05.732 ****** 2020-10-15T16:35:05.801 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:35:05.803 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdZupzHZUu6WNiQyvAcbfv06vYa6pi8acR'], 'uuids': ['a951d209-77ba-4dc3-b487-4f92a7694431']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.173005", "end": "2020-10-15 16:35:05.535396", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdZupzHZUu6WNiQyvAcbfv06vYa6pi8acR"], "labels": [], "masters": [], "uuids": ["a951d209-77ba-4dc3-b487-4f92a7694431"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:05.362391", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.168356 s, 685 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.168356 s, 685 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:05.813 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:05.824 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.074 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-10-15T16:35:06.075 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-10-15T16:35:06.108 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:06 +0000 (0:00:00.334) 0:02:06.066 ****** 2020-10-15T16:35:06.115 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:35:06.117 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubddHfioysUgiTVKiZeJuJ04o74nvsp245a'], 'uuids': ['1f15b17b-5d26-487a-80d5-1e0375235256']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.171261", "end": "2020-10-15 16:35:05.848311", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubddHfioysUgiTVKiZeJuJ04o74nvsp245a"], "labels": [], "masters": [], "uuids": ["1f15b17b-5d26-487a-80d5-1e0375235256"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:05.677050", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.166333 s, 693 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.166333 s, 693 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:06.118 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D', 'wwn-0x5000c500871b5cd5'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J33D-part1', 'wwn-0x5000c500871b5cd5-part1'], 'uuids': ['6a302439-42e1-4f94-b095-dec7b770edc2']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b5cd5', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J33D', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D", "wwn-0x5000c500871b5cd5"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J33D-part1", "wwn-0x5000c500871b5cd5-part1"], "labels": [], "masters": [], "uuids": ["6a302439-42e1-4f94-b095-dec7b770edc2"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J33D", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b5cd5"}}, "skip_reason": 2020-10-15T16:35:06.120 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-15T16:35:06.129 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-10-15T16:35:06.131 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.162 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:06 +0000 (0:00:00.054) 0:02:06.121 ****** 2020-10-15T16:35:06.292 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:35:06.294 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-X9k38g-df55-vY91-KhVT-0Z2j-S0tH-rqmlv0', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054966", "end": "2020-10-15 16:35:06.029009", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-X9k38g-df55-vY91-KhVT-0Z2j-S0tH-rqmlv0", "nvme-INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:05.974043", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0500742 s, 2.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0500742 s, 2.3 GB 2020-10-15T16:35:06.296 INFO:teuthology.task.ansible.out:/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:06.304 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.316 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.358 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-10-15T16:35:06.359 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-10-15 16:32:58 UTC", "ActiveEnterTimestampMonotonic": "26627151", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service systemd-journald.socket basic.target network.target sysinit.target system.slice -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-10-15 16:32:51 UTC", "AssertTimestampMonotonic": "20188088", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-10-15 16:32:51 UTC", "ConditionTimestampMonotonic": "20188087", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-10-15T16:35:06.361 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1876", "ExecMainStartTimestamp": "Thu 2020-10-15 16:32:58 UTC", "ExecMainStartTimestampMonotonic": "26627104", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-10-15 16:32:51 UTC", "InactiveExitTimestampMonotonic": "20190281", "InvocationID": "8f63ab11a23343199a5ffbaafb7bbfd8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-10-15T16:35:06.362 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1876", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-10-15 16:32:58 UTC", "S 2020-10-15T16:35:06.364 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26627151", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-10-15 16:32:58 UTC", "WatchdogTimestampMonotonic": "26627148", "WatchdogUSec": "0"}} 2020-10-15T16:35:06.393 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:06 +0000 (0:00:00.230) 0:02:06.352 ****** 2020-10-15T16:35:06.420 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:06.421 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-10-15T16:35:06.453 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:06 +0000 (0:00:00.059) 0:02:06.412 ****** 2020-10-15T16:35:06.480 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:06.481 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.516 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:06 +0000 (0:00:00.062) 0:02:06.474 ****** 2020-10-15T16:35:06.542 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-10-15T16:35:06.544 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.575 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:06 +0000 (0:00:00.059) 0:02:06.534 ****** 2020-10-15T16:35:06.602 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-15T16:35:06.604 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdpExDw5LWvsChoEPkj3wTPc8ZSFKkI3Un'], 'uuids': ['05528fd3-9a0e-467d-8715-130a109ccba3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.163461", "end": "2020-10-15 16:35:06.338051", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdpExDw5LWvsChoEPkj3wTPc8ZSFKkI3Un"], "labels": [], "masters": [], "uuids": ["05528fd3-9a0e-467d-8715-130a109ccba3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:06.174590", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158772 s, 726 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158772 s, 726 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:06.614 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.626 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:06.915 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4Cubddfkmifup8n8Y2yklK0MiAPDqHT7TMPLI'], 'uuids': ['619b5798-65c4-4cc2-ac34-2ef0cfa39202']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.168818", "end": "2020-10-15 16:35:06.650542", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4Cubddfkmifup8n8Y2yklK0MiAPDqHT7TMPLI"], "labels": [], "masters": [], "uuids": ["619b5798-65c4-4cc2-ac34-2ef0cfa39202"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:06.481724", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.163896 s, 704 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.163896 s, 704 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:06.927 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:07.209 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdrMkvwWTV5sX2685bKoiVNG0vwDAjAKWB'], 'uuids': ['1db01386-704c-4ada-b277-492a1d9c8c17']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.159051", "end": "2020-10-15 16:35:06.942474", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OeoJHEEXCEzwFuwgJ1xQ68nqdIx4CubdrMkvwWTV5sX2685bKoiVNG0vwDAjAKWB"], "labels": [], "masters": [], "uuids": ["1db01386-704c-4ada-b277-492a1d9c8c17"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-15 16:35:06.783423", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154211 s, 748 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154211 s, 748 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:07.253 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.678) 0:02:07.212 ****** 2020-10-15T16:35:07.396 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-10-15T16:35:07.398 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"cache_update_time": 1602779376, "cache_updated": false, "changed": false} 2020-10-15T16:35:07.432 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.178) 0:02:07.390 ****** 2020-10-15T16:35:07.460 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-10-15T16:35:07.462 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:07.494 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.061) 0:02:07.452 ****** 2020-10-15T16:35:07.552 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-10-15T16:35:07.553 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.159778", "end": "2020-10-15 16:35:07.285827", "rc": 0, "start": "2020-10-15 16:35:07.126049", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:07.587 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.092) 0:02:07.545 ****** 2020-10-15T16:35:07.626 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-10-15T16:35:07.629 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-10-15T16:35:07.661 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.074) 0:02:07.620 ****** 2020-10-15T16:35:07.783 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-10-15T16:35:07.785 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.063433", "end": "2020-10-15 16:35:07.525364", "rc": 0, "start": "2020-10-15 16:35:07.461931", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:07.818 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.156) 0:02:07.776 ****** 2020-10-15T16:35:07.844 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-10-15T16:35:07.881 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.063) 0:02:07.839 ****** 2020-10-15T16:35:07.917 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-10-15T16:35:07.919 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:07.957 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:07 +0000 (0:00:00.076) 0:02:07.916 ****** 2020-10-15T16:35:07.985 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.018 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.061) 0:02:07.977 ****** 2020-10-15T16:35:08.025 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-10-15T16:35:08.027 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-10-15T16:35:08.059 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.041) 0:02:08.018 ****** 2020-10-15T16:35:08.065 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-10-15T16:35:08.067 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.103 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.043) 0:02:08.062 ****** 2020-10-15T16:35:08.132 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-10-15T16:35:08.134 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.170 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.066) 0:02:08.129 ****** 2020-10-15T16:35:08.305 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-10-15T16:35:08.307 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-15T16:35:08.339 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.168) 0:02:08.297 ****** 2020-10-15T16:35:08.377 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-15T16:35:08.378 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'block_used': 13749371, 'uuid': '6a302439-42e1-4f94-b095-dec7b770edc2', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226345737, 'size_available': 927112138752, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149456, 'block_size': 4096, 'inode_available': 60905520}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226345737, "block_size": 4096, "block_total": 240095108, "block_used": 13749371, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905520, "inode_total": 61054976, "inode_used": 149456, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927112138752, "size_total": 983429562368, "uuid": "6a302439-42e1-4f94-b095-dec7b770edc2"}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.410 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.071) 0:02:08.369 ****** 2020-10-15T16:35:08.437 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-10-15T16:35:08.439 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.471 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.060) 0:02:08.429 ****** 2020-10-15T16:35:08.497 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:08.499 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.530 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.059) 0:02:08.489 ****** 2020-10-15T16:35:08.557 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:08.559 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.590 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.059) 0:02:08.549 ****** 2020-10-15T16:35:08.617 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-10-15T16:35:08.619 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.650 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.059) 0:02:08.609 ****** 2020-10-15T16:35:08.687 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-10-15T16:35:08.689 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.728 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.077) 0:02:08.686 ****** 2020-10-15T16:35:08.756 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-10-15T16:35:08.757 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:08.792 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:08 +0000 (0:00:00.064) 0:02:08.750 ****** 2020-10-15T16:35:09.049 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-10-15T16:35:09.051 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-10-15T16:35:09.086 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:09 +0000 (0:00:00.294) 0:02:09.045 ****** 2020-10-15T16:35:09.104 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-10-15T16:35:09.106 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-10-15T16:35:09.139 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:09 +0000 (0:00:00.053) 0:02:09.098 ****** 2020-10-15T16:35:09.578 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-10-15T16:35:09.580 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false} 2020-10-15T16:35:09.614 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:09 +0000 (0:00:00.475) 0:02:09.573 ****** 2020-10-15T16:35:09.642 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-10-15T16:35:09.643 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-10-15T16:35:09.675 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:09 +0000 (0:00:00.060) 0:02:09.633 ****** 2020-10-15T16:35:09.816 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-10-15T16:35:09.818 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1602779364.027754, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1602779364.0237541, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 111543, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1602779364.0237541, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3466128932", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-10-15T16:35:09.851 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:09 +0000 (0:00:00.176) 0:02:09.810 ****** 2020-10-15T16:35:10.230 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-10-15T16:35:10.232 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-10-15T16:35:10.263 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.411) 0:02:10.222 ****** 2020-10-15T16:35:10.290 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-10-15T16:35:10.292 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-10-15T16:35:10.324 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.061) 0:02:10.283 ****** 2020-10-15T16:35:10.352 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-10-15T16:35:10.354 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.385 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.060) 0:02:10.344 ****** 2020-10-15T16:35:10.459 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:35:10.461 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:10.522 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-10-15T16:35:10.524 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-10-15T16:35:10.556 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.170) 0:02:10.514 ****** 2020-10-15T16:35:10.676 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-10-15T16:35:10.678 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-10-15T16:35:10.718 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.161) 0:02:10.676 ****** 2020-10-15T16:35:10.727 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:35:10.729 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:10.750 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-10-15T16:35:10.752 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.802 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.084) 0:02:10.760 ****** 2020-10-15T16:35:10.840 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-10-15T16:35:10.842 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.851 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.853 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.886 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.083) 0:02:10.844 ****** 2020-10-15T16:35:10.923 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-10-15T16:35:10.924 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi183.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.934 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.936 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.946 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:10.980 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:10 +0000 (0:00:00.094) 0:02:10.938 ****** 2020-10-15T16:35:10.987 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:35:10.988 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:11.009 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:11.011 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.046 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.066) 0:02:11.005 ****** 2020-10-15T16:35:11.073 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-10-15T16:35:11.075 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.114 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.068) 0:02:11.073 ****** 2020-10-15T16:35:11.142 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.179 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.065) 0:02:11.138 ****** 2020-10-15T16:35:11.207 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-10-15T16:35:11.209 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.244 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.064) 0:02:11.202 ****** 2020-10-15T16:35:11.250 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:35:11.252 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:11.273 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-10-15T16:35:11.275 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.315 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.066) 0:02:11.269 ****** 2020-10-15T16:35:11.355 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-10-15T16:35:11.357 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.414 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.103) 0:02:11.373 ****** 2020-10-15T16:35:11.443 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-10-15T16:35:11.445 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.479 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.064) 0:02:11.437 ****** 2020-10-15T16:35:11.485 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-15T16:35:11.487 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-15T16:35:11.524 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.044) 0:02:11.482 ****** 2020-10-15T16:35:11.530 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-10-15T16:35:11.532 INFO:teuthology.task.ansible.out:skipping: [smithi183.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:11.567 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.043) 0:02:11.525 ****** 2020-10-15T16:35:11.667 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-10-15T16:35:11.669 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-10-15T16:35:11.704 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:11 +0000 (0:00:00.136) 0:02:11.662 ****** 2020-10-15T16:35:11.722 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-10-15T16:35:11.723 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010933", "end": "2020-10-15 16:35:11.828225", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-10-15 16:35:11.817292", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:11.848 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006079", "end": "2020-10-15 16:35:11.950659", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-10-15 16:35:11.944580", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:11.850 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:11.953 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-10-15T16:35:11.955 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005348", "end": "2020-10-15 16:35:12.066563", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-10-15 16:35:12.061215", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:11.964 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-15T16:35:11.966 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:12.079 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:12.090 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-10-15T16:35:12.092 INFO:teuthology.task.ansible.out:ok: [smithi183.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.019445", "end": "2020-10-15 16:35:12.196219", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-10-15 16:35:12.176774", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:12.126 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:12 +0000 (0:00:00.422) 0:02:12.085 ****** 2020-10-15T16:35:12.153 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-10-15T16:35:12.187 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:12 +0000 (0:00:00.060) 0:02:12.146 ****** 2020-10-15T16:35:12.205 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-15T16:35:12.207 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-15T16:35:12.217 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:12.256 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:12 +0000 (0:00:00.068) 0:02:12.214 ****** 2020-10-15T16:35:12.315 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-10-15T16:35:12.317 INFO:teuthology.task.ansible.out:changed: [smithi183.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-10-15T16:35:12.393 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-10-15T16:35:12.395 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-10-15T16:35:12.427 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:12 +0000 (0:00:00.171) 0:02:12.385 ****** 2020-10-15T16:35:12.468 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-10-15T16:35:12.470 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-10-15T16:35:12.507 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:12 +0000 (0:00:00.080) 0:02:12.466 ****** 2020-10-15T16:35:12.896 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-10-15T16:35:12.968 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true} Thursday 15 October 2020 16:35:12 +0000 (0:00:00.423) 0:02:12.889 ****** 2020-10-15T16:35:13.146 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-10-15T16:35:13.147 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-10-15T16:35:13.181 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:13 +0000 (0:00:00.249) 0:02:13.139 ****** 2020-10-15T16:35:13.323 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-10-15T16:35:13.325 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-10-15T16:35:13.357 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:13 +0000 (0:00:00.176) 0:02:13.316 ****** 2020-10-15T16:35:13.490 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-10-15T16:35:13.492 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-10-15T16:35:13.524 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:13 +0000 (0:00:00.166) 0:02:13.482 ****** 2020-10-15T16:35:13.860 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-10-15T16:35:13.862 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-10-15T16:35:13.894 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:13 +0000 (0:00:00.370) 0:02:13.853 ****** 2020-10-15T16:35:14.213 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-10-15T16:35:14.215 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-10-15T16:35:14.262 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:14 +0000 (0:00:00.367) 0:02:14.221 ****** 2020-10-15T16:35:15.181 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-10-15T16:35:15.183 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:35:15.214 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:15 +0000 (0:00:00.951) 0:02:15.172 ****** 2020-10-15T16:35:15.396 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-10-15T16:35:15.397 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048988", "end": "2020-10-15 16:35:15.138053", "rc": 0, "start": "2020-10-15 16:35:15.089065", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-10-15T16:35:15.430 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:15 +0000 (0:00:00.215) 0:02:15.388 ****** 2020-10-15T16:35:15.457 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-10-15T16:35:15.458 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:15.490 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:15 +0000 (0:00:00.060) 0:02:15.448 ****** 2020-10-15T16:35:15.516 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-10-15T16:35:15.518 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:15.549 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:15 +0000 (0:00:00.059) 0:02:15.508 ****** 2020-10-15T16:35:16.425 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-10-15T16:35:16.427 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:35:16.459 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:16 +0000 (0:00:00.909) 0:02:16.417 ****** 2020-10-15T16:35:16.487 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:16.520 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:16 +0000 (0:00:00.061) 0:02:16.479 ****** 2020-10-15T16:35:16.835 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-10-15T16:35:16.837 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-10-15T16:35:16.869 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:16 +0000 (0:00:00.348) 0:02:16.827 ****** 2020-10-15T16:35:16.896 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-10-15T16:35:16.898 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:16.932 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:16 +0000 (0:00:00.063) 0:02:16.891 ****** 2020-10-15T16:35:17.145 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-10-15T16:35:17.146 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-10-15 16:32:53 UTC", "ActiveEnterTimestampMonotonic": "23970021", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount network.target sysinit.target systemd-journald.socket systemd-tmpfiles-setup.service basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-10-15 16:32:47 UTC", "AssertTimestampMonotonic": "17492112", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-10-15 16:32:47 UTC", "ConditionTimestampMonotonic": "17492112", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-10-15T16:35:17.148 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1911", "ExecMainStartTimestamp": "Thu 2020-10-15 16:32:53 UTC", "ExecMainStartTimestampMonotonic": "23969976", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-10-15 16:32:47 UTC", "InactiveExitTimestampMonotonic": "17493152", "InvocationID": "4f3d8827f0054c1ea5fd21179ee396a9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-10-15T16:35:17.149 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1911", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-10-15 16:32:53 UTC", "S 2020-10-15T16:35:17.151 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23970021", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-10-15 16:32:53 UTC", "WatchdogTimestampMonotonic": "23970019", "WatchdogUSec": "0"}} 2020-10-15T16:35:17.181 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:17 +0000 (0:00:00.248) 0:02:17.139 ****** 2020-10-15T16:35:17.208 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:17.209 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-10-15T16:35:17.241 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:17 +0000 (0:00:00.060) 0:02:17.199 ****** 2020-10-15T16:35:17.268 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-15T16:35:17.269 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:17.301 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:17 +0000 (0:00:00.060) 0:02:17.260 ****** 2020-10-15T16:35:17.328 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-10-15T16:35:17.330 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:17.361 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:17 +0000 (0:00:00.060) 0:02:17.320 ****** 2020-10-15T16:35:18.282 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-10-15T16:35:18.284 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"cache_update_time": 1602779421, "cache_updated": false, "changed": false} 2020-10-15T16:35:18.316 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:18 +0000 (0:00:00.954) 0:02:18.274 ****** 2020-10-15T16:35:18.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-10-15T16:35:18.345 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:18.377 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:18 +0000 (0:00:00.061) 0:02:18.335 ****** 2020-10-15T16:35:18.507 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-10-15T16:35:18.509 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-10-15T16:35:18.540 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:18 +0000 (0:00:00.163) 0:02:18.499 ****** 2020-10-15T16:35:18.846 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-10-15T16:35:18.848 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-10-15T16:35:18.883 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:18 +0000 (0:00:00.342) 0:02:18.842 ****** 2020-10-15T16:35:19.653 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-10-15T16:35:19.655 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-10-15T16:35:19.685 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:19 +0000 (0:00:00.801) 0:02:19.643 ****** 2020-10-15T16:35:20.194 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-10-15T16:35:20.195 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false} 2020-10-15T16:35:20.227 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.541) 0:02:20.185 ****** 2020-10-15T16:35:20.253 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-10-15T16:35:20.255 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-10-15T16:35:20.286 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.059) 0:02:20.245 ****** 2020-10-15T16:35:20.426 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-10-15T16:35:20.428 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1602779341.3325236, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1602779341.3285217, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 111543, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1602779341.3285217, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3466128932", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-10-15T16:35:20.459 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.173) 0:02:20.418 ****** 2020-10-15T16:35:20.589 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-10-15T16:35:20.591 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-10-15T16:35:20.622 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.162) 0:02:20.581 ****** 2020-10-15T16:35:20.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi038.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-10-15T16:35:20.682 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.059) 0:02:20.641 ****** 2020-10-15T16:35:20.710 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-10-15T16:35:20.711 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:20.743 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.060) 0:02:20.701 ****** 2020-10-15T16:35:20.873 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-10-15T16:35:20.875 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-10-15T16:35:20.907 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:20 +0000 (0:00:00.164) 0:02:20.865 ****** 2020-10-15T16:35:21.049 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-10-15T16:35:21.051 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-10-15T16:35:21.085 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.178) 0:02:21.044 ****** 2020-10-15T16:35:21.112 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-10-15T16:35:21.114 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.164 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.078) 0:02:21.122 ****** 2020-10-15T16:35:21.204 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-10-15T16:35:21.206 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.215 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.217 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.249 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.085) 0:02:21.208 ****** 2020-10-15T16:35:21.287 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-10-15T16:35:21.288 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.298 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.300 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.310 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.347 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.097) 0:02:21.305 ****** 2020-10-15T16:35:21.374 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.408 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.061) 0:02:21.367 ****** 2020-10-15T16:35:21.435 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-10-15T16:35:21.437 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.468 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.059) 0:02:21.427 ****** 2020-10-15T16:35:21.495 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-10-15T16:35:21.497 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.530 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.062) 0:02:21.489 ****** 2020-10-15T16:35:21.557 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-10-15T16:35:21.559 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.590 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.059) 0:02:21.548 ****** 2020-10-15T16:35:21.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-10-15T16:35:21.618 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.650 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.060) 0:02:21.609 ****** 2020-10-15T16:35:21.677 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-10-15T16:35:21.679 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.712 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.061) 0:02:21.670 ****** 2020-10-15T16:35:21.749 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-10-15T16:35:21.751 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.787 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.074) 0:02:21.745 ****** 2020-10-15T16:35:21.814 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-10-15T16:35:21.816 INFO:teuthology.task.ansible.out:skipping: [smithi038.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-15T16:35:21.847 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:21 +0000 (0:00:00.059) 0:02:21.805 ****** 2020-10-15T16:35:22.007 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-10-15T16:35:22.009 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.007298", "end": "2020-10-15 16:35:21.739113", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-10-15 16:35:21.731815", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:22.132 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006193", "end": "2020-10-15 16:35:21.875038", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-10-15 16:35:21.868845", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:22.245 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005579", "end": "2020-10-15 16:35:21.992461", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-10-15 16:35:21.986882", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:22.389 INFO:teuthology.task.ansible.out:ok: [smithi038.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.020922", "end": "2020-10-15 16:35:22.128311", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-10-15 16:35:22.107389", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-15T16:35:22.424 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:22 +0000 (0:00:00.577) 0:02:22.383 ****** 2020-10-15T16:35:22.451 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-10-15T16:35:22.484 INFO:teuthology.task.ansible.out:Thursday 15 October 2020 16:35:22 +0000 (0:00:00.059) 0:02:22.443 ****** 2020-10-15T16:35:22.613 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-10-15T16:35:22.615 INFO:teuthology.task.ansible.out:changed: [smithi038.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-10-15T16:35:22.705 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-10-15T16:35:22.708 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-10-15T16:35:22.710 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-10-15T16:35:22.711 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-10-15T16:35:22.713 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi038.front.sepia.ceph.com : ok=114 changed=28 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 smithi183.front.sepia.ceph.com : ok=114 changed=28 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 Thursday 15 October 2020 16:35:22 +0000 (0:00:00.227) 0:02:22.670 ****** =============================================================================== users : Create all admin users with sudo access. ----------------------- 17.18s users : Update authorized_keys using the keys repo ---------------------- 6.32s testnode : Update apt cache. -------------------------------------------- 3.02s testnode : Install apt keys --------------------------------------------- 2.01s Gathering Facts --------------------------------------------------------- 1.73s testnode : Install non aarch64 packages --------------------------------- 0.97s testnode : Install python-apt ------------------------------------------- 0.97s testnode : Install python-pip on apt based systems. --------------------- 0.95s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.95s testnode : Install ntp package on apt based systems. -------------------- 0.91s common : Set the hardware clock ----------------------------------------- 0.88s users : Clone the keys repo --------------------------------------------- 0.81s testnode : Install packages via pip ------------------------------------- 0.80s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.77s Gathering Facts --------------------------------------------------------- 0.77s ansible-managed : Add authorized keys for the ansible user. ------------- 0.59s testnode : List any leftover Ceph artifacts from previous jobs ---------- 0.58s ansible-managed : Create the ansible user. ------------------------------ 0.55s testnode : Purge r 2020-10-15T16:35:22.714 INFO:teuthology.task.ansible.out:esolvconf --------------------------------------------- 0.54s testnode : Add ssh pubkeys ---------------------------------------------- 0.54s 2020-10-15T16:35:23.031 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi183.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com')] 2020-10-15T16:35:23.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:35:23.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:35:23.110 INFO:teuthology.orchestra.run.smithi183:> true 2020-10-15T16:35:23.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-15T16:35:23.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-15T16:35:23.396 INFO:teuthology.orchestra.run.smithi038:> true 2020-10-15T16:35:23.599 INFO:teuthology.run_tasks:Running task clock... 2020-10-15T16:35:23.611 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-10-15T16:35:23.612 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-10-15T16:35:23.612 INFO:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-10-15T16:35:23.683 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-10-15T16:35:23.684 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Command line: ntpd -gq 2020-10-15T16:35:23.684 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: proto: precision = 0.114 usec (-23) 2020-10-15T16:35:23.685 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-15T16:35:23.685 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: restrict ::: KOD does nothing without LIMITED. 2020-10-15T16:35:23.685 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listen and drop on 0 v6wildcard [::]:123 2020-10-15T16:35:23.685 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-10-15T16:35:23.685 INFO:teuthology.orchestra.run.smithi183.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-15T16:35:23.685 INFO:teuthology.orchestra.run.smithi183.stderr:restrict ::: KOD does nothing without LIMITED. 2020-10-15T16:35:23.686 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listen normally on 2 lo 127.0.0.1:123 2020-10-15T16:35:23.686 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listen normally on 3 enp3s0f1 172.21.15.183:123 2020-10-15T16:35:23.686 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listen normally on 4 lo [::1]:123 2020-10-15T16:35:23.687 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8085%5]:123 2020-10-15T16:35:23.687 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:23 ntpd[7795]: Listening on routing socket on fd #22 for interface updates 2020-10-15T16:35:24.685 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:24.685 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:24.685 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:24.685 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:24.685 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:24.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:24.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:24.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:24 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:26.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:26.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:26.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:26.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:26.686 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:26.687 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:26.687 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:26.687 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:26 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:28.687 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:28.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:28.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:28.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:28.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:28.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:28.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:28.689 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:28.690 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:28.690 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:28.690 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:28 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:30.687 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:30 ntpd[7795]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:30.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:30 ntpd[7795]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:30.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:30 ntpd[7795]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:30.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:30 ntpd[7795]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:30.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:30 ntpd[7795]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-10-15T16:35:30.688 INFO:teuthology.orchestra.run.smithi183.stderr:15 Oct 16:35:30 ntpd[7795]: can't open /var/log/ntpstats/loopstats.20201015: Permission denied 2020-10-15T16:35:30.689 INFO:teuthology.orchestra.run.smithi183.stdout:15 Oct 16:35:30 ntpd[7795]: ntpd: time set -0.124526 s 2020-10-15T16:35:30.689 INFO:teuthology.orchestra.run.smithi183.stdout:ntpd: time set -0.124526s 2020-10-15T16:35:30.760 INFO:teuthology.orchestra.run.smithi183.stdout: remote refid st t when poll reach delay offset jitter 2020-10-15T16:35:30.760 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================== 2020-10-15T16:35:30.760 INFO:teuthology.orchestra.run.smithi183.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:30.760 INFO:teuthology.orchestra.run.smithi183.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:30.760 INFO:teuthology.orchestra.run.smithi183.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:30.761 INFO:teuthology.orchestra.run.smithi183.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:30.761 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-10-15T16:35:30.762 INFO:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-10-15T16:35:30.830 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-10-15T16:35:30.831 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Command line: ntpd -gq 2020-10-15T16:35:30.831 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: proto: precision = 0.127 usec (-23) 2020-10-15T16:35:30.832 INFO:teuthology.orchestra.run.smithi038.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-15T16:35:30.832 INFO:teuthology.orchestra.run.smithi038.stderr:restrict ::: KOD does nothing without LIMITED. 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: restrict ::: KOD does nothing without LIMITED. 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listen and drop on 0 v6wildcard [::]:123 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listen normally on 2 lo 127.0.0.1:123 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listen normally on 3 ens1f0 172.21.15.38:123 2020-10-15T16:35:30.833 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listen normally on 4 lo [::1]:123 2020-10-15T16:35:30.834 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15fc%3]:123 2020-10-15T16:35:30.834 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:30 ntpd[7873]: Listening on routing socket on fd #22 for interface updates 2020-10-15T16:35:31.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:31 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:31.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:31 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:31.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:31 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:31.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:31 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:33 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:34.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:34 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:34.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:34 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:34.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:34 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:34.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:34 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:35.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:35.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:35.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:35.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:35.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:35.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:35 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:36.828 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:36 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:36.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:36 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:36.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:36 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:36.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:36 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:37.828 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:37.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:37.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:37.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:37.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:37.829 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:37.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:37.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:37.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-15T16:35:37.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/rawstats.20201015: Permission denied 2020-10-15T16:35:37.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-15T16:35:37.830 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/peerstats.20201015: Permission denied 2020-10-15T16:35:37.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-10-15T16:35:37.831 INFO:teuthology.orchestra.run.smithi038.stderr:15 Oct 16:35:37 ntpd[7873]: can't open /var/log/ntpstats/loopstats.20201015: Permission denied 2020-10-15T16:35:37.831 INFO:teuthology.orchestra.run.smithi038.stdout:15 Oct 16:35:37 ntpd[7873]: ntpd: time set +0.223726 s 2020-10-15T16:35:37.831 INFO:teuthology.orchestra.run.smithi038.stdout:ntpd: time set +0.223726s 2020-10-15T16:35:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: remote refid st t when poll reach delay offset jitter 2020-10-15T16:35:37.878 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================== 2020-10-15T16:35:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-15T16:35:37.880 INFO:teuthology.run_tasks:Running task install... 2020-10-15T16:35:37.888 DEBUG:teuthology.task.install:project ceph 2020-10-15T16:35:37.888 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1'}} 2020-10-15T16:35:37.889 DEBUG:teuthology.task.install:config {'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1'} 2020-10-15T16:35:37.889 INFO:teuthology.task.install:Using flavor: basic 2020-10-15T16:35:37.895 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-10-15T16:35:37.896 INFO:teuthology.task.install:extra packages: [] 2020-10-15T16:35:37.896 INFO:teuthology.orchestra.run.smithi183:> sudo lsb_release -is 2020-10-15T16:35:37.950 INFO:teuthology.orchestra.run.smithi183.stdout:Ubuntu 2020-10-15T16:35:37.955 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-15T16:35:37.955 INFO:teuthology.orchestra.run.smithi038:> sudo lsb_release -is 2020-10-15T16:35:37.956 INFO:teuthology.orchestra.run.smithi183:> sudo apt-key list | grep Ceph 2020-10-15T16:35:38.010 INFO:teuthology.orchestra.run.smithi038.stdout:Ubuntu 2020-10-15T16:35:38.016 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-15T16:35:38.016 INFO:teuthology.orchestra.run.smithi038:> sudo apt-key list | grep Ceph 2020-10-15T16:35:38.044 INFO:teuthology.orchestra.run.smithi183.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-10-15T16:35:38.104 INFO:teuthology.orchestra.run.smithi183.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-10-15T16:35:38.104 INFO:teuthology.orchestra.run.smithi183.stdout:uid [ unknown] Ceph.com (release key) 2020-10-15T16:35:38.105 INFO:teuthology.orchestra.run.smithi183:> sudo lsb_release -is 2020-10-15T16:35:38.114 INFO:teuthology.orchestra.run.smithi038.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-10-15T16:35:38.156 INFO:teuthology.orchestra.run.smithi183.stdout:Ubuntu 2020-10-15T16:35:38.161 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-15T16:35:38.161 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-10-15T16:35:38.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 2020-10-15T16:35:38.172 INFO:teuthology.orchestra.run.smithi038.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-10-15T16:35:38.173 INFO:teuthology.orchestra.run.smithi038.stdout:uid [ unknown] Ceph.com (release key) 2020-10-15T16:35:38.173 INFO:teuthology.orchestra.run.smithi038:> sudo lsb_release -is 2020-10-15T16:35:38.224 INFO:teuthology.orchestra.run.smithi038.stdout:Ubuntu 2020-10-15T16:35:38.229 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-15T16:35:38.230 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-10-15T16:35:38.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 2020-10-15T16:35:38.328 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default/ 2020-10-15T16:35:38.328 INFO:teuthology.task.install.deb:Package version is 16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:35:38.389 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default/ 2020-10-15T16:35:38.389 INFO:teuthology.task.install.deb:Package version is 16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:35:38.458 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:35:38.459 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-15T16:35:38.464 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-15T16:35:38.473 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:35:38.473 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:35:38.473 INFO:teuthology.orchestra.run.smithi183.stderr:154 bytes copied, 6.6189e-05 s, 2.3 MB/s 2020-10-15T16:35:38.474 INFO:teuthology.orchestra.run.smithi183:> sudo apt-get update 2020-10-15T16:35:38.514 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:35:38.514 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-15T16:35:38.519 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-15T16:35:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:35:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:35:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:154 bytes copied, 6.8747e-05 s, 2.2 MB/s 2020-10-15T16:35:38.528 INFO:teuthology.orchestra.run.smithi038:> sudo apt-get update 2020-10-15T16:35:38.671 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-10-15T16:35:38.744 INFO:teuthology.orchestra.run.smithi183.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic InRelease 2020-10-15T16:35:38.768 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-10-15T16:35:38.781 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-15T16:35:38.793 INFO:teuthology.orchestra.run.smithi183.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic Release.gpg 2020-10-15T16:35:38.797 INFO:teuthology.orchestra.run.smithi038.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic InRelease 2020-10-15T16:35:38.821 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-10-15T16:35:38.822 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,950 B] 2020-10-15T16:35:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-10-15T16:35:38.839 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-15T16:35:38.846 INFO:teuthology.orchestra.run.smithi038.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic Release.gpg 2020-10-15T16:35:38.861 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-10-15T16:35:38.864 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-10-15T16:35:38.869 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,950 B] 2020-10-15T16:35:38.906 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-10-15T16:35:38.911 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-10-15T16:35:38.941 INFO:teuthology.orchestra.run.smithi183.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-10-15T16:35:38.991 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-10-15T16:35:39.267 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 26.7 kB in 1s (45.1 kB/s) 2020-10-15T16:35:39.457 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 26.7 kB in 1s (37.0 kB/s) 2020-10-15T16:35:40.484 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:35:40.530 INFO:teuthology.orchestra.run.smithi183:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-6334-g417a7e57-1bionic cephadm=16.0.0-6334-g417a7e57-1bionic ceph-mds=16.0.0-6334-g417a7e57-1bionic ceph-mgr=16.0.0-6334-g417a7e57-1bionic ceph-common=16.0.0-6334-g417a7e57-1bionic ceph-fuse=16.0.0-6334-g417a7e57-1bionic ceph-test=16.0.0-6334-g417a7e57-1bionic radosgw=16.0.0-6334-g417a7e57-1bionic python3-rados=16.0.0-6334-g417a7e57-1bionic python3-rgw=16.0.0-6334-g417a7e57-1bionic python3-cephfs=16.0.0-6334-g417a7e57-1bionic python3-rbd=16.0.0-6334-g417a7e57-1bionic libcephfs2=16.0.0-6334-g417a7e57-1bionic libcephfs-dev=16.0.0-6334-g417a7e57-1bionic librados2=16.0.0-6334-g417a7e57-1bionic librbd1=16.0.0-6334-g417a7e57-1bionic rbd-fuse=16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:35:40.608 INFO:teuthology.orchestra.run.smithi183.stdout:Reading package lists... 2020-10-15T16:35:40.721 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:35:40.771 INFO:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-6334-g417a7e57-1bionic cephadm=16.0.0-6334-g417a7e57-1bionic ceph-mds=16.0.0-6334-g417a7e57-1bionic ceph-mgr=16.0.0-6334-g417a7e57-1bionic ceph-common=16.0.0-6334-g417a7e57-1bionic ceph-fuse=16.0.0-6334-g417a7e57-1bionic ceph-test=16.0.0-6334-g417a7e57-1bionic radosgw=16.0.0-6334-g417a7e57-1bionic python3-rados=16.0.0-6334-g417a7e57-1bionic python3-rgw=16.0.0-6334-g417a7e57-1bionic python3-cephfs=16.0.0-6334-g417a7e57-1bionic python3-rbd=16.0.0-6334-g417a7e57-1bionic libcephfs2=16.0.0-6334-g417a7e57-1bionic libcephfs-dev=16.0.0-6334-g417a7e57-1bionic librados2=16.0.0-6334-g417a7e57-1bionic librbd1=16.0.0-6334-g417a7e57-1bionic rbd-fuse=16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:35:40.836 INFO:teuthology.orchestra.run.smithi183.stdout:Building dependency tree... 2020-10-15T16:35:40.837 INFO:teuthology.orchestra.run.smithi183.stdout:Reading state information... 2020-10-15T16:35:40.849 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2020-10-15T16:35:41.000 INFO:teuthology.orchestra.run.smithi183.stdout:The following additional packages will be installed: 2020-10-15T16:35:41.001 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-10-15T16:35:41.002 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-10-15T16:35:41.002 INFO:teuthology.orchestra.run.smithi183.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-10-15T16:35:41.002 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-10-15T16:35:41.002 INFO:teuthology.orchestra.run.smithi183.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-10-15T16:35:41.002 INFO:teuthology.orchestra.run.smithi183.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-10-15T16:35:41.002 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-10-15T16:35:41.003 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-10-15T16:35:41.003 INFO:teuthology.orchestra.run.smithi183.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-10-15T16:35:41.003 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-10-15T16:35:41.003 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-10-15T16:35:41.003 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-10-15T16:35:41.003 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-15T16:35:41.004 INFO:teuthology.orchestra.run.smithi183.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-15T16:35:41.005 INFO:teuthology.orchestra.run.smithi183.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-15T16:35:41.005 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-10-15T16:35:41.005 INFO:teuthology.orchestra.run.smithi183.stdout:Suggested packages: 2020-10-15T16:35:41.005 INFO:teuthology.orchestra.run.smithi183.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-10-15T16:35:41.005 INFO:teuthology.orchestra.run.smithi183.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-10-15T16:35:41.006 INFO:teuthology.orchestra.run.smithi183.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-10-15T16:35:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-10-15T16:35:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-10-15T16:35:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-10-15T16:35:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-10-15T16:35:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-10-15T16:35:41.007 INFO:teuthology.orchestra.run.smithi183.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-10-15T16:35:41.085 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2020-10-15T16:35:41.086 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2020-10-15T16:35:41.106 INFO:teuthology.orchestra.run.smithi183.stdout:The following NEW packages will be installed: 2020-10-15T16:35:41.107 INFO:teuthology.orchestra.run.smithi183.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-10-15T16:35:41.107 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-10-15T16:35:41.108 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-10-15T16:35:41.108 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-10-15T16:35:41.108 INFO:teuthology.orchestra.run.smithi183.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-10-15T16:35:41.108 INFO:teuthology.orchestra.run.smithi183.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-10-15T16:35:41.109 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-10-15T16:35:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-15T16:35:41.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-15T16:35:41.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-15T16:35:41.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-15T16:35:41.111 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-10-15T16:35:41.111 INFO:teuthology.orchestra.run.smithi183.stdout:The following packages will be upgraded: 2020-10-15T16:35:41.112 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 librbd1 2020-10-15T16:35:41.267 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2020-10-15T16:35:41.267 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-10-15T16:35:41.268 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-10-15T16:35:41.268 INFO:teuthology.orchestra.run.smithi038.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-10-15T16:35:41.268 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-10-15T16:35:41.269 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-10-15T16:35:41.270 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-10-15T16:35:41.270 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-10-15T16:35:41.270 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-15T16:35:41.271 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-15T16:35:41.272 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-10-15T16:35:41.272 INFO:teuthology.orchestra.run.smithi183.stdout:2 upgraded, 110 newly installed, 0 to remove and 18 not upgraded. 2020-10-15T16:35:41.272 INFO:teuthology.orchestra.run.smithi183.stdout:Need to get 168 MB of archives. 2020-10-15T16:35:41.273 INFO:teuthology.orchestra.run.smithi183.stdout:After this operation, 780 MB of additional disk space will be used. 2020-10-15T16:35:41.273 INFO:teuthology.orchestra.run.smithi183.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-6334-g417a7e57-1bionic [1,866 kB] 2020-10-15T16:35:41.273 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2020-10-15T16:35:41.273 INFO:teuthology.orchestra.run.smithi038.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-10-15T16:35:41.273 INFO:teuthology.orchestra.run.smithi038.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-10-15T16:35:41.273 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-10-15T16:35:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-10-15T16:35:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-10-15T16:35:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-10-15T16:35:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-10-15T16:35:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-10-15T16:35:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-10-15T16:35:41.297 INFO:teuthology.orchestra.run.smithi183.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-10-15T16:35:41.377 INFO:teuthology.orchestra.run.smithi183.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-10-15T16:35:41.377 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2020-10-15T16:35:41.377 INFO:teuthology.orchestra.run.smithi038.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-10-15T16:35:41.377 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-10-15T16:35:41.378 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-10-15T16:35:41.378 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-10-15T16:35:41.378 INFO:teuthology.orchestra.run.smithi038.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-10-15T16:35:41.379 INFO:teuthology.orchestra.run.smithi038.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-10-15T16:35:41.379 INFO:teuthology.orchestra.run.smithi038.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-10-15T16:35:41.379 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-10-15T16:35:41.380 INFO:teuthology.orchestra.run.smithi038.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-15T16:35:41.381 INFO:teuthology.orchestra.run.smithi038.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-15T16:35:41.382 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-15T16:35:41.382 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-10-15T16:35:41.382 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages will be upgraded: 2020-10-15T16:35:41.382 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 librbd1 2020-10-15T16:35:41.453 INFO:teuthology.orchestra.run.smithi183.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-6334-g417a7e57-1bionic [3,172 kB] 2020-10-15T16:35:41.548 INFO:teuthology.orchestra.run.smithi183.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-6334-g417a7e57-1bionic [474 kB] 2020-10-15T16:35:41.564 INFO:teuthology.orchestra.run.smithi183.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-6334-g417a7e57-1bionic [327 kB] 2020-10-15T16:35:41.569 INFO:teuthology.orchestra.run.smithi038.stdout:2 upgraded, 110 newly installed, 0 to remove and 18 not upgraded. 2020-10-15T16:35:41.569 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 168 MB of archives. 2020-10-15T16:35:41.569 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 780 MB of additional disk space will be used. 2020-10-15T16:35:41.569 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-10-15T16:35:41.574 INFO:teuthology.orchestra.run.smithi183.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-6334-g417a7e57-1bionic [20.6 kB] 2020-10-15T16:35:41.575 INFO:teuthology.orchestra.run.smithi183.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-6334-g417a7e57-1bionic [168 kB] 2020-10-15T16:35:41.580 INFO:teuthology.orchestra.run.smithi183.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-6334-g417a7e57-1bionic [27.7 kB] 2020-10-15T16:35:41.581 INFO:teuthology.orchestra.run.smithi183.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-6334-g417a7e57-1bionic [312 kB] 2020-10-15T16:35:41.592 INFO:teuthology.orchestra.run.smithi183.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-6334-g417a7e57-1bionic [3,233 kB] 2020-10-15T16:35:41.630 INFO:teuthology.orchestra.run.smithi183.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-10-15T16:35:41.637 INFO:teuthology.orchestra.run.smithi183.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-10-15T16:35:41.639 INFO:teuthology.orchestra.run.smithi183.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-10-15T16:35:41.649 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-10-15T16:35:41.670 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-6334-g417a7e57-1bionic [1,866 kB] 2020-10-15T16:35:41.699 INFO:teuthology.orchestra.run.smithi183.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-6334-g417a7e57-1bionic [99.3 kB] 2020-10-15T16:35:41.702 INFO:teuthology.orchestra.run.smithi183.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-6334-g417a7e57-1bionic [414 kB] 2020-10-15T16:35:41.717 INFO:teuthology.orchestra.run.smithi183.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-6334-g417a7e57-1bionic [20.4 MB] 2020-10-15T16:35:41.743 INFO:teuthology.orchestra.run.smithi183.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-10-15T16:35:41.746 INFO:teuthology.orchestra.run.smithi183.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-10-15T16:35:41.782 INFO:teuthology.orchestra.run.smithi183.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-10-15T16:35:41.790 INFO:teuthology.orchestra.run.smithi183.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-10-15T16:35:41.799 INFO:teuthology.orchestra.run.smithi183.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-10-15T16:35:41.811 INFO:teuthology.orchestra.run.smithi183.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-10-15T16:35:41.821 INFO:teuthology.orchestra.run.smithi183.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-10-15T16:35:41.846 INFO:teuthology.orchestra.run.smithi183.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-10-15T16:35:41.849 INFO:teuthology.orchestra.run.smithi183.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-10-15T16:35:41.869 INFO:teuthology.orchestra.run.smithi183.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-10-15T16:35:41.880 INFO:teuthology.orchestra.run.smithi183.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-10-15T16:35:41.881 INFO:teuthology.orchestra.run.smithi183.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-10-15T16:35:41.896 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-10-15T16:35:41.908 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-10-15T16:35:41.921 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-10-15T16:35:41.948 INFO:teuthology.orchestra.run.smithi183.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-10-15T16:35:41.948 INFO:teuthology.orchestra.run.smithi183.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-10-15T16:35:41.950 INFO:teuthology.orchestra.run.smithi183.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-10-15T16:35:41.951 INFO:teuthology.orchestra.run.smithi183.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-10-15T16:35:41.951 INFO:teuthology.orchestra.run.smithi183.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-10-15T16:35:41.954 INFO:teuthology.orchestra.run.smithi183.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-10-15T16:35:41.958 INFO:teuthology.orchestra.run.smithi183.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-10-15T16:35:42.021 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-10-15T16:35:42.025 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-10-15T16:35:42.026 INFO:teuthology.orchestra.run.smithi183.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-10-15T16:35:42.027 INFO:teuthology.orchestra.run.smithi183.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-10-15T16:35:42.048 INFO:teuthology.orchestra.run.smithi183.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-10-15T16:35:42.058 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-10-15T16:35:42.064 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-10-15T16:35:42.073 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-10-15T16:35:42.084 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-10-15T16:35:42.092 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-10-15T16:35:42.105 INFO:teuthology.orchestra.run.smithi183.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-10-15T16:35:42.106 INFO:teuthology.orchestra.run.smithi183.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-10-15T16:35:42.107 INFO:teuthology.orchestra.run.smithi183.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-10-15T16:35:42.111 INFO:teuthology.orchestra.run.smithi183.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-10-15T16:35:42.113 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-10-15T16:35:42.117 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-10-15T16:35:42.119 INFO:teuthology.orchestra.run.smithi183.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-10-15T16:35:42.123 INFO:teuthology.orchestra.run.smithi183.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-10-15T16:35:42.123 INFO:teuthology.orchestra.run.smithi183.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-10-15T16:35:42.135 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-10-15T16:35:42.148 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-10-15T16:35:42.184 INFO:teuthology.orchestra.run.smithi183.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-10-15T16:35:42.214 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-10-15T16:35:42.215 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-10-15T16:35:42.216 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-10-15T16:35:42.218 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-10-15T16:35:42.219 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-10-15T16:35:42.219 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-10-15T16:35:42.222 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-10-15T16:35:42.225 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-10-15T16:35:42.277 INFO:teuthology.orchestra.run.smithi183.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-10-15T16:35:42.277 INFO:teuthology.orchestra.run.smithi183.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-10-15T16:35:42.293 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-10-15T16:35:42.294 INFO:teuthology.orchestra.run.smithi038.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-10-15T16:35:42.372 INFO:teuthology.orchestra.run.smithi038.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-10-15T16:35:42.373 INFO:teuthology.orchestra.run.smithi038.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-10-15T16:35:42.373 INFO:teuthology.orchestra.run.smithi038.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-10-15T16:35:42.374 INFO:teuthology.orchestra.run.smithi038.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-10-15T16:35:42.378 INFO:teuthology.orchestra.run.smithi038.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-10-15T16:35:42.386 INFO:teuthology.orchestra.run.smithi038.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-10-15T16:35:42.390 INFO:teuthology.orchestra.run.smithi038.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-10-15T16:35:42.391 INFO:teuthology.orchestra.run.smithi038.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-10-15T16:35:42.424 INFO:teuthology.orchestra.run.smithi183.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-6334-g417a7e57-1bionic [5,630 kB] 2020-10-15T16:35:42.451 INFO:teuthology.orchestra.run.smithi038.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-10-15T16:35:42.536 INFO:teuthology.orchestra.run.smithi038.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-10-15T16:35:42.537 INFO:teuthology.orchestra.run.smithi038.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-10-15T16:35:42.550 INFO:teuthology.orchestra.run.smithi183.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-10-15T16:35:42.610 INFO:teuthology.orchestra.run.smithi183.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-10-15T16:35:42.612 INFO:teuthology.orchestra.run.smithi183.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-10-15T16:35:42.616 INFO:teuthology.orchestra.run.smithi183.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-6334-g417a7e57-1bionic [161 kB] 2020-10-15T16:35:42.620 INFO:teuthology.orchestra.run.smithi183.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-6334-g417a7e57-1bionic [1,377 kB] 2020-10-15T16:35:42.666 INFO:teuthology.orchestra.run.smithi183.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-6334-g417a7e57-1bionic [6,649 kB] 2020-10-15T16:35:42.713 INFO:teuthology.orchestra.run.smithi183.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-10-15T16:35:42.714 INFO:teuthology.orchestra.run.smithi183.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-10-15T16:35:42.714 INFO:teuthology.orchestra.run.smithi183.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-10-15T16:35:42.714 INFO:teuthology.orchestra.run.smithi183.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-10-15T16:35:42.715 INFO:teuthology.orchestra.run.smithi183.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-10-15T16:35:42.715 INFO:teuthology.orchestra.run.smithi183.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-10-15T16:35:42.716 INFO:teuthology.orchestra.run.smithi183.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-10-15T16:35:42.722 INFO:teuthology.orchestra.run.smithi183.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-10-15T16:35:42.723 INFO:teuthology.orchestra.run.smithi183.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-10-15T16:35:42.805 INFO:teuthology.orchestra.run.smithi183.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-10-15T16:35:42.805 INFO:teuthology.orchestra.run.smithi183.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-10-15T16:35:42.809 INFO:teuthology.orchestra.run.smithi183.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-10-15T16:35:42.810 INFO:teuthology.orchestra.run.smithi183.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-10-15T16:35:42.810 INFO:teuthology.orchestra.run.smithi183.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-10-15T16:35:42.812 INFO:teuthology.orchestra.run.smithi183.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-10-15T16:35:42.815 INFO:teuthology.orchestra.run.smithi183.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-10-15T16:35:42.816 INFO:teuthology.orchestra.run.smithi183.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-10-15T16:35:42.819 INFO:teuthology.orchestra.run.smithi183.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-10-15T16:35:42.882 INFO:teuthology.orchestra.run.smithi038.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-10-15T16:35:42.883 INFO:teuthology.orchestra.run.smithi183.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-10-15T16:35:42.895 INFO:teuthology.orchestra.run.smithi183.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-6334-g417a7e57-1bionic [24.6 MB] 2020-10-15T16:35:42.901 INFO:teuthology.orchestra.run.smithi183.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-10-15T16:35:42.903 INFO:teuthology.orchestra.run.smithi183.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-10-15T16:35:42.904 INFO:teuthology.orchestra.run.smithi183.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-10-15T16:35:42.904 INFO:teuthology.orchestra.run.smithi183.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-10-15T16:35:42.904 INFO:teuthology.orchestra.run.smithi183.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-10-15T16:35:42.905 INFO:teuthology.orchestra.run.smithi183.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-10-15T16:35:42.906 INFO:teuthology.orchestra.run.smithi183.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-10-15T16:35:42.907 INFO:teuthology.orchestra.run.smithi183.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-10-15T16:35:42.940 INFO:teuthology.orchestra.run.smithi038.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-10-15T16:35:42.941 INFO:teuthology.orchestra.run.smithi038.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-10-15T16:35:42.962 INFO:teuthology.orchestra.run.smithi183.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-10-15T16:35:42.964 INFO:teuthology.orchestra.run.smithi038.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-10-15T16:35:42.965 INFO:teuthology.orchestra.run.smithi038.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-10-15T16:35:42.965 INFO:teuthology.orchestra.run.smithi038.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-10-15T16:35:42.967 INFO:teuthology.orchestra.run.smithi038.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-10-15T16:35:42.968 INFO:teuthology.orchestra.run.smithi038.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-10-15T16:35:42.969 INFO:teuthology.orchestra.run.smithi038.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-10-15T16:35:42.970 INFO:teuthology.orchestra.run.smithi038.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-10-15T16:35:42.986 INFO:teuthology.orchestra.run.smithi183.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-10-15T16:35:42.989 INFO:teuthology.orchestra.run.smithi183.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-10-15T16:35:43.010 INFO:teuthology.orchestra.run.smithi183.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-10-15T16:35:43.010 INFO:teuthology.orchestra.run.smithi183.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-10-15T16:35:43.010 INFO:teuthology.orchestra.run.smithi183.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-10-15T16:35:43.044 INFO:teuthology.orchestra.run.smithi038.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-10-15T16:35:43.046 INFO:teuthology.orchestra.run.smithi038.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-10-15T16:35:43.124 INFO:teuthology.orchestra.run.smithi038.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-10-15T16:35:43.126 INFO:teuthology.orchestra.run.smithi038.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-10-15T16:35:43.131 INFO:teuthology.orchestra.run.smithi038.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-10-15T16:35:43.132 INFO:teuthology.orchestra.run.smithi183.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-10-15T16:35:43.132 INFO:teuthology.orchestra.run.smithi038.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-10-15T16:35:43.133 INFO:teuthology.orchestra.run.smithi183.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-10-15T16:35:43.133 INFO:teuthology.orchestra.run.smithi038.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-10-15T16:35:43.134 INFO:teuthology.orchestra.run.smithi183.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-10-15T16:35:43.136 INFO:teuthology.orchestra.run.smithi038.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-10-15T16:35:43.137 INFO:teuthology.orchestra.run.smithi038.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-10-15T16:35:43.138 INFO:teuthology.orchestra.run.smithi038.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-10-15T16:35:43.139 INFO:teuthology.orchestra.run.smithi183.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-10-15T16:35:43.139 INFO:teuthology.orchestra.run.smithi183.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-10-15T16:35:43.140 INFO:teuthology.orchestra.run.smithi183.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-10-15T16:35:43.140 INFO:teuthology.orchestra.run.smithi183.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-10-15T16:35:43.142 INFO:teuthology.orchestra.run.smithi183.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-10-15T16:35:43.142 INFO:teuthology.orchestra.run.smithi183.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-10-15T16:35:43.193 INFO:teuthology.orchestra.run.smithi038.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-6334-g417a7e57-1bionic [3,172 kB] 2020-10-15T16:35:43.203 INFO:teuthology.orchestra.run.smithi038.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-10-15T16:35:43.209 INFO:teuthology.orchestra.run.smithi038.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-10-15T16:35:43.282 INFO:teuthology.orchestra.run.smithi038.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-10-15T16:35:43.290 INFO:teuthology.orchestra.run.smithi038.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-10-15T16:35:43.290 INFO:teuthology.orchestra.run.smithi038.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-10-15T16:35:43.291 INFO:teuthology.orchestra.run.smithi038.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-10-15T16:35:43.292 INFO:teuthology.orchestra.run.smithi038.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-10-15T16:35:43.294 INFO:teuthology.orchestra.run.smithi038.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-10-15T16:35:43.299 INFO:teuthology.orchestra.run.smithi038.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-10-15T16:35:43.299 INFO:teuthology.orchestra.run.smithi038.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-10-15T16:35:43.361 INFO:teuthology.orchestra.run.smithi038.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-10-15T16:35:43.362 INFO:teuthology.orchestra.run.smithi038.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-10-15T16:35:43.440 INFO:teuthology.orchestra.run.smithi038.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-10-15T16:35:43.461 INFO:teuthology.orchestra.run.smithi038.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-10-15T16:35:43.462 INFO:teuthology.orchestra.run.smithi038.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-10-15T16:35:43.462 INFO:teuthology.orchestra.run.smithi038.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-10-15T16:35:43.554 INFO:teuthology.orchestra.run.smithi038.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-10-15T16:35:43.554 INFO:teuthology.orchestra.run.smithi038.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-10-15T16:35:43.599 INFO:teuthology.orchestra.run.smithi038.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-10-15T16:35:43.609 INFO:teuthology.orchestra.run.smithi038.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-10-15T16:35:43.609 INFO:teuthology.orchestra.run.smithi038.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-10-15T16:35:43.609 INFO:teuthology.orchestra.run.smithi038.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-10-15T16:35:43.610 INFO:teuthology.orchestra.run.smithi038.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-10-15T16:35:43.612 INFO:teuthology.orchestra.run.smithi038.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-10-15T16:35:43.612 INFO:teuthology.orchestra.run.smithi038.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-10-15T16:35:43.740 INFO:teuthology.orchestra.run.smithi183.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-6334-g417a7e57-1bionic [3,808 B] 2020-10-15T16:35:43.740 INFO:teuthology.orchestra.run.smithi183.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-6334-g417a7e57-1bionic [559 kB] 2020-10-15T16:35:43.757 INFO:teuthology.orchestra.run.smithi183.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-6334-g417a7e57-1bionic [1,976 kB] 2020-10-15T16:35:43.826 INFO:teuthology.orchestra.run.smithi183.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-6334-g417a7e57-1bionic [51.3 kB] 2020-10-15T16:35:43.827 INFO:teuthology.orchestra.run.smithi183.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-6334-g417a7e57-1bionic [55.9 kB] 2020-10-15T16:35:43.828 INFO:teuthology.orchestra.run.smithi183.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-6334-g417a7e57-1bionic [2,862 kB] 2020-10-15T16:35:43.927 INFO:teuthology.orchestra.run.smithi183.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-6334-g417a7e57-1bionic [7,883 kB] 2020-10-15T16:35:44.198 INFO:teuthology.orchestra.run.smithi183.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-6334-g417a7e57-1bionic [3,928 B] 2020-10-15T16:35:44.199 INFO:teuthology.orchestra.run.smithi183.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-6334-g417a7e57-1bionic [22.3 kB] 2020-10-15T16:35:44.199 INFO:teuthology.orchestra.run.smithi183.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-6334-g417a7e57-1bionic [49.5 MB] 2020-10-15T16:35:45.509 INFO:teuthology.orchestra.run.smithi038.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-6334-g417a7e57-1bionic [474 kB] 2020-10-15T16:35:45.864 INFO:teuthology.orchestra.run.smithi038.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-6334-g417a7e57-1bionic [327 kB] 2020-10-15T16:35:45.902 INFO:teuthology.orchestra.run.smithi183.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-6334-g417a7e57-1bionic [19.2 kB] 2020-10-15T16:35:45.903 INFO:teuthology.orchestra.run.smithi183.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-6334-g417a7e57-1bionic [9,779 kB] 2020-10-15T16:35:46.121 INFO:teuthology.orchestra.run.smithi038.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-6334-g417a7e57-1bionic [20.6 kB] 2020-10-15T16:35:46.128 INFO:teuthology.orchestra.run.smithi038.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-6334-g417a7e57-1bionic [168 kB] 2020-10-15T16:35:46.214 INFO:teuthology.orchestra.run.smithi038.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-6334-g417a7e57-1bionic [27.7 kB] 2020-10-15T16:35:46.234 INFO:teuthology.orchestra.run.smithi183.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-6334-g417a7e57-1bionic [76.9 kB] 2020-10-15T16:35:46.235 INFO:teuthology.orchestra.run.smithi038.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-6334-g417a7e57-1bionic [312 kB] 2020-10-15T16:35:46.302 INFO:teuthology.orchestra.run.smithi038.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-6334-g417a7e57-1bionic [3,233 kB] 2020-10-15T16:35:46.895 INFO:teuthology.orchestra.run.smithi038.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-6334-g417a7e57-1bionic [99.3 kB] 2020-10-15T16:35:46.916 INFO:teuthology.orchestra.run.smithi038.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-6334-g417a7e57-1bionic [414 kB] 2020-10-15T16:35:46.934 INFO:teuthology.orchestra.run.smithi183.stdout:Fetched 168 MB in 5s (32.9 MB/s) 2020-10-15T16:35:46.984 INFO:teuthology.orchestra.run.smithi038.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-6334-g417a7e57-1bionic [20.4 MB] 2020-10-15T16:35:47.056 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-uritemplate. 2020-10-15T16:35:47.104 INFO:teuthology.orchestra.run.smithi183.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137592 files and directories currently installed.) 2020-10-15T16:35:47.108 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-10-15T16:35:47.146 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-10-15T16:35:47.482 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-10-15T16:35:47.496 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-10-15T16:35:47.582 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-15T16:35:47.941 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../002-librbd1_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:48.069 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librbd1 (16.0.0-6334-g417a7e57-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-15T16:35:48.569 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../003-librados2_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:48.690 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librados2 (16.0.0-6334-g417a7e57-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-15T16:35:49.327 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libcephfs2. 2020-10-15T16:35:49.342 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:49.385 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libcephfs2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:49.620 INFO:teuthology.orchestra.run.smithi038.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-6334-g417a7e57-1bionic [5,630 kB] 2020-10-15T16:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rados. 2020-10-15T16:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../005-python3-rados_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rados (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:50.090 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-10-15T16:35:50.106 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:35:50.149 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-ceph-argparse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:50.186 INFO:teuthology.orchestra.run.smithi038.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-6334-g417a7e57-1bionic [161 kB] 2020-10-15T16:35:50.202 INFO:teuthology.orchestra.run.smithi038.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-6334-g417a7e57-1bionic [1,377 kB] 2020-10-15T16:35:50.337 INFO:teuthology.orchestra.run.smithi038.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-6334-g417a7e57-1bionic [6,649 kB] 2020-10-15T16:35:50.451 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cephfs. 2020-10-15T16:35:50.467 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:50.510 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cephfs (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:50.837 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-ceph-common. 2020-10-15T16:35:50.852 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:35:50.896 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:50.944 INFO:teuthology.orchestra.run.smithi038.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-6334-g417a7e57-1bionic [24.6 MB] 2020-10-15T16:35:51.190 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-prettytable. 2020-10-15T16:35:51.206 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-10-15T16:35:51.241 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-10-15T16:35:51.451 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rbd. 2020-10-15T16:35:51.465 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:51.510 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rbd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:51.870 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-10-15T16:35:51.884 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-10-15T16:35:51.929 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-15T16:35:52.315 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-10-15T16:35:52.329 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-10-15T16:35:52.375 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-15T16:35:52.659 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package librgw2. 2020-10-15T16:35:52.673 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../013-librgw2_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:52.709 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking librgw2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:52.888 INFO:teuthology.orchestra.run.smithi038.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-6334-g417a7e57-1bionic [3,808 B] 2020-10-15T16:35:52.889 INFO:teuthology.orchestra.run.smithi038.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-6334-g417a7e57-1bionic [559 kB] 2020-10-15T16:35:52.926 INFO:teuthology.orchestra.run.smithi038.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-6334-g417a7e57-1bionic [1,976 kB] 2020-10-15T16:35:53.066 INFO:teuthology.orchestra.run.smithi038.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-6334-g417a7e57-1bionic [51.3 kB] 2020-10-15T16:35:53.072 INFO:teuthology.orchestra.run.smithi038.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-6334-g417a7e57-1bionic [55.9 kB] 2020-10-15T16:35:53.073 INFO:teuthology.orchestra.run.smithi038.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-6334-g417a7e57-1bionic [2,862 kB] 2020-10-15T16:35:53.275 INFO:teuthology.orchestra.run.smithi038.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-6334-g417a7e57-1bionic [7,883 kB] 2020-10-15T16:35:53.278 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rgw. 2020-10-15T16:35:53.292 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:53.362 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:53.689 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-10-15T16:35:53.704 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-10-15T16:35:53.748 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-15T16:35:53.803 INFO:teuthology.orchestra.run.smithi038.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-6334-g417a7e57-1bionic [3,928 B] 2020-10-15T16:35:53.803 INFO:teuthology.orchestra.run.smithi038.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-6334-g417a7e57-1bionic [22.3 kB] 2020-10-15T16:35:53.804 INFO:teuthology.orchestra.run.smithi038.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-6334-g417a7e57-1bionic [49.5 MB] 2020-10-15T16:35:53.992 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-10-15T16:35:54.009 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-10-15T16:35:54.042 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-10-15T16:35:54.411 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package liboath0. 2020-10-15T16:35:54.428 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-10-15T16:35:54.470 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-10-15T16:35:54.789 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libradosstriper1. 2020-10-15T16:35:54.805 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:54.848 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libradosstriper1 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:55.267 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-common. 2020-10-15T16:35:55.281 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../019-ceph-common_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:55.317 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:56.226 INFO:teuthology.orchestra.run.smithi038.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-6334-g417a7e57-1bionic [19.2 kB] 2020-10-15T16:35:56.226 INFO:teuthology.orchestra.run.smithi038.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-6334-g417a7e57-1bionic [9,779 kB] 2020-10-15T16:35:56.563 INFO:teuthology.orchestra.run.smithi038.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuval-cls-fifo/417a7e57ff62f4d3de1c307caeb5ea5d04efadf1/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-6334-g417a7e57-1bionic [76.9 kB] 2020-10-15T16:35:57.215 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-base. 2020-10-15T16:35:57.228 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../020-ceph-base_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:57.281 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-base (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:57.285 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 168 MB in 15s (11.1 MB/s) 2020-10-15T16:35:57.514 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-uritemplate. 2020-10-15T16:35:57.564 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137592 files and directories currently installed.) 2020-10-15T16:35:57.568 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-10-15T16:35:57.604 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-10-15T16:35:57.899 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-10-15T16:35:57.913 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-10-15T16:35:57.991 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-15T16:35:58.244 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-dateutil. 2020-10-15T16:35:58.259 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-10-15T16:35:58.302 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-10-15T16:35:58.335 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../002-librbd1_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:58.462 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librbd1 (16.0.0-6334-g417a7e57-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-15T16:35:58.613 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-asn1crypto. 2020-10-15T16:35:58.629 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-10-15T16:35:58.671 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-10-15T16:35:58.973 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../003-librados2_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:58.991 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cffi-backend. 2020-10-15T16:35:59.005 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-10-15T16:35:59.049 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-10-15T16:35:59.108 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librados2 (16.0.0-6334-g417a7e57-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-15T16:35:59.394 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cryptography. 2020-10-15T16:35:59.409 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-10-15T16:35:59.452 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-15T16:35:59.745 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libcephfs2. 2020-10-15T16:35:59.761 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:35:59.803 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libcephfs2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:35:59.940 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-openssl. 2020-10-15T16:35:59.953 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-10-15T16:36:00.014 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-10-15T16:36:00.089 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rados. 2020-10-15T16:36:00.102 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../005-python3-rados_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:00.140 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rados (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:00.276 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-10-15T16:36:00.288 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:00.325 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-modules-core (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:00.484 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-10-15T16:36:00.499 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:00.542 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-ceph-argparse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:00.679 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-bcrypt. 2020-10-15T16:36:00.693 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-10-15T16:36:00.729 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-10-15T16:36:00.837 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cephfs. 2020-10-15T16:36:00.853 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:00.895 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cephfs (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:01.032 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cherrypy3. 2020-10-15T16:36:01.047 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-10-15T16:36:01.099 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-10-15T16:36:01.223 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-ceph-common. 2020-10-15T16:36:01.237 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:01.281 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:01.543 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-prettytable. 2020-10-15T16:36:01.552 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jwt. 2020-10-15T16:36:01.556 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-10-15T16:36:01.569 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-10-15T16:36:01.585 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-10-15T16:36:01.611 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-10-15T16:36:01.804 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rbd. 2020-10-15T16:36:01.820 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:01.862 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rbd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:01.905 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-logutils. 2020-10-15T16:36:01.921 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-10-15T16:36:01.963 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-10-15T16:36:02.257 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-10-15T16:36:02.272 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-10-15T16:36:02.291 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-markupsafe. 2020-10-15T16:36:02.307 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-10-15T16:36:02.315 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-15T16:36:02.350 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-10-15T16:36:02.652 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-mako. 2020-10-15T16:36:02.660 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-10-15T16:36:02.667 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-10-15T16:36:02.675 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-10-15T16:36:02.710 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-15T16:36:02.710 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-10-15T16:36:03.013 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librgw2. 2020-10-15T16:36:03.028 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../013-librgw2_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:03.038 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-simplegeneric. 2020-10-15T16:36:03.052 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-10-15T16:36:03.071 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librgw2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:03.105 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-10-15T16:36:03.441 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-singledispatch. 2020-10-15T16:36:03.460 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-10-15T16:36:03.500 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-10-15T16:36:03.665 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rgw. 2020-10-15T16:36:03.679 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:03.725 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:03.802 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-webob. 2020-10-15T16:36:03.816 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-10-15T16:36:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-10-15T16:36:04.043 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-10-15T16:36:04.058 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-10-15T16:36:04.093 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-15T16:36:04.171 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-bs4. 2020-10-15T16:36:04.186 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-10-15T16:36:04.230 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-10-15T16:36:04.329 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-10-15T16:36:04.344 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-10-15T16:36:04.379 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-10-15T16:36:04.566 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-waitress. 2020-10-15T16:36:04.580 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-10-15T16:36:04.625 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-10-15T16:36:04.740 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liboath0. 2020-10-15T16:36:04.757 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-10-15T16:36:04.799 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-10-15T16:36:04.961 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-tempita. 2020-10-15T16:36:04.976 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-10-15T16:36:05.027 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-10-15T16:36:05.135 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libradosstriper1. 2020-10-15T16:36:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:05.194 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libradosstriper1 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:05.338 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-paste. 2020-10-15T16:36:05.352 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-10-15T16:36:05.405 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-15T16:36:05.580 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-common. 2020-10-15T16:36:05.597 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../019-ceph-common_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:05.638 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:05.752 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-10-15T16:36:05.768 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-10-15T16:36:05.819 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-10-15T16:36:06.121 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pastedeploy. 2020-10-15T16:36:06.135 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-10-15T16:36:06.179 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-10-15T16:36:06.499 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-webtest. 2020-10-15T16:36:06.512 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-10-15T16:36:06.549 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-10-15T16:36:06.918 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pecan. 2020-10-15T16:36:06.933 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-10-15T16:36:06.977 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-10-15T16:36:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-werkzeug. 2020-10-15T16:36:07.345 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-10-15T16:36:07.390 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-15T16:36:07.511 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-base. 2020-10-15T16:36:07.525 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../020-ceph-base_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:07.579 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-base (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:07.785 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr. 2020-10-15T16:36:07.798 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:07.835 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:08.321 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mon. 2020-10-15T16:36:08.336 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:08.379 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mon (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:08.566 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-dateutil. 2020-10-15T16:36:08.580 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-10-15T16:36:08.625 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-10-15T16:36:08.986 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-asn1crypto. 2020-10-15T16:36:09.000 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-10-15T16:36:09.044 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-10-15T16:36:09.289 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cffi-backend. 2020-10-15T16:36:09.290 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-osd. 2020-10-15T16:36:09.302 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-10-15T16:36:09.306 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:09.330 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-10-15T16:36:09.349 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-osd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:09.675 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cryptography. 2020-10-15T16:36:09.689 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-10-15T16:36:09.733 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-15T16:36:10.179 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-openssl. 2020-10-15T16:36:10.193 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-10-15T16:36:10.237 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-10-15T16:36:10.490 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-10-15T16:36:10.504 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:10.565 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-modules-core (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:11.367 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph. 2020-10-15T16:36:11.376 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../048-ceph_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:11.421 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:11.712 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-bcrypt. 2020-10-15T16:36:11.727 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-10-15T16:36:11.770 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-10-15T16:36:12.056 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cherrypy3. 2020-10-15T16:36:12.072 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-10-15T16:36:12.115 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-10-15T16:36:12.460 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jwt. 2020-10-15T16:36:12.465 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-fuse. 2020-10-15T16:36:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-10-15T16:36:12.480 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:12.518 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-10-15T16:36:12.523 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:12.846 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-logutils. 2020-10-15T16:36:12.860 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-10-15T16:36:12.905 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-10-15T16:36:12.927 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mds. 2020-10-15T16:36:12.940 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:12.985 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mds (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:13.233 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-markupsafe. 2020-10-15T16:36:13.247 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-10-15T16:36:13.291 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-10-15T16:36:13.496 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package cephadm. 2020-10-15T16:36:13.511 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../051-cephadm_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:13.554 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:13.569 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-mako. 2020-10-15T16:36:13.585 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-10-15T16:36:13.627 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-10-15T16:36:13.857 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jinja2. 2020-10-15T16:36:13.873 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-10-15T16:36:13.915 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-15T16:36:13.939 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-simplegeneric. 2020-10-15T16:36:13.952 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-10-15T16:36:13.997 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-10-15T16:36:14.243 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-10-15T16:36:14.259 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:14.302 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:14.333 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-singledispatch. 2020-10-15T16:36:14.348 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-10-15T16:36:14.400 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-10-15T16:36:14.638 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-repoze.lru. 2020-10-15T16:36:14.652 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-10-15T16:36:14.688 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-10-15T16:36:14.711 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webob. 2020-10-15T16:36:14.725 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-10-15T16:36:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-10-15T16:36:14.991 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-routes. 2020-10-15T16:36:15.005 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-10-15T16:36:15.049 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-10-15T16:36:15.072 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-bs4. 2020-10-15T16:36:15.086 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-10-15T16:36:15.122 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-10-15T16:36:15.360 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-10-15T16:36:15.374 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:15.411 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-dashboard (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:15.451 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-waitress. 2020-10-15T16:36:15.465 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-10-15T16:36:15.509 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-10-15T16:36:15.862 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tempita. 2020-10-15T16:36:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-10-15T16:36:15.921 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-10-15T16:36:16.257 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-paste. 2020-10-15T16:36:16.272 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-10-15T16:36:16.302 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-numpy. 2020-10-15T16:36:16.316 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-10-15T16:36:16.323 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-15T16:36:16.351 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-15T16:36:16.627 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-10-15T16:36:16.641 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-10-15T16:36:16.685 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-10-15T16:36:16.965 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-decorator. 2020-10-15T16:36:16.980 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-10-15T16:36:17.021 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pastedeploy. 2020-10-15T16:36:17.023 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-10-15T16:36:17.036 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-10-15T16:36:17.079 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-10-15T16:36:17.343 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-scipy. 2020-10-15T16:36:17.357 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-10-15T16:36:17.402 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-10-15T16:36:17.407 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webtest. 2020-10-15T16:36:17.421 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-10-15T16:36:17.466 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-10-15T16:36:17.811 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pecan. 2020-10-15T16:36:17.824 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-10-15T16:36:17.869 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-10-15T16:36:18.224 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-werkzeug. 2020-10-15T16:36:18.239 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-10-15T16:36:18.282 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-15T16:36:18.519 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-10-15T16:36:18.533 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-10-15T16:36:18.586 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-10-15T16:36:18.629 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr. 2020-10-15T16:36:18.645 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:18.712 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:19.072 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-joblib. 2020-10-15T16:36:19.087 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-10-15T16:36:19.130 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-joblib (0.11-1) ... 2020-10-15T16:36:19.224 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mon. 2020-10-15T16:36:19.239 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:19.282 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mon (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:19.475 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-sklearn. 2020-10-15T16:36:19.489 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-10-15T16:36:19.533 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-10-15T16:36:20.090 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-10-15T16:36:20.104 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:20.148 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:20.278 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-osd. 2020-10-15T16:36:20.292 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:20.336 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-osd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:21.268 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-monotonic. 2020-10-15T16:36:21.282 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-10-15T16:36:21.326 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-10-15T16:36:21.629 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-fasteners. 2020-10-15T16:36:21.643 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-10-15T16:36:21.687 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-10-15T16:36:22.032 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-httplib2. 2020-10-15T16:36:22.047 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-10-15T16:36:22.090 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-15T16:36:22.383 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph. 2020-10-15T16:36:22.398 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../048-ceph_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:22.418 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyasn1. 2020-10-15T16:36:22.432 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-10-15T16:36:22.441 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:22.485 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-10-15T16:36:22.761 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-fuse. 2020-10-15T16:36:22.777 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:22.819 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:22.838 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-10-15T16:36:22.852 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-10-15T16:36:22.896 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-15T16:36:23.215 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-rsa. 2020-10-15T16:36:23.231 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mds. 2020-10-15T16:36:23.231 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-10-15T16:36:23.248 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:23.282 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-10-15T16:36:23.314 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mds (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:23.594 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-10-15T16:36:23.609 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-10-15T16:36:23.644 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-15T16:36:23.826 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package cephadm. 2020-10-15T16:36:23.842 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../051-cephadm_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:23.884 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:24.090 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-oauth2client. 2020-10-15T16:36:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-10-15T16:36:24.148 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-10-15T16:36:24.204 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jinja2. 2020-10-15T16:36:24.218 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-10-15T16:36:24.262 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-15T16:36:24.510 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-setuptools. 2020-10-15T16:36:24.524 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-10-15T16:36:24.568 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-10-15T16:36:24.623 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-10-15T16:36:24.639 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:24.682 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:24.979 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-websocket. 2020-10-15T16:36:24.995 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-10-15T16:36:25.035 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-repoze.lru. 2020-10-15T16:36:25.037 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-10-15T16:36:25.049 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-10-15T16:36:25.110 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-10-15T16:36:25.332 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-kubernetes. 2020-10-15T16:36:25.349 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-10-15T16:36:25.390 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-15T16:36:25.413 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-routes. 2020-10-15T16:36:25.428 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-10-15T16:36:25.471 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-10-15T16:36:25.783 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-10-15T16:36:25.797 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:25.820 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-10-15T16:36:25.835 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:25.841 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-dashboard (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:25.878 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:26.197 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-json-pointer. 2020-10-15T16:36:26.214 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-10-15T16:36:26.255 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-10-15T16:36:26.633 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-jsonpatch. 2020-10-15T16:36:26.648 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-10-15T16:36:26.695 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-15T16:36:26.758 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-numpy. 2020-10-15T16:36:26.773 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-10-15T16:36:26.817 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-15T16:36:27.012 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-10-15T16:36:27.027 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:27.406 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-decorator. 2020-10-15T16:36:27.420 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-10-15T16:36:27.463 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-mgr-rook (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:27.464 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-10-15T16:36:27.814 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libonig4:amd64. 2020-10-15T16:36:27.826 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-scipy. 2020-10-15T16:36:27.828 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-10-15T16:36:27.840 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-10-15T16:36:27.881 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-10-15T16:36:27.892 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-10-15T16:36:28.242 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libjq1:amd64. 2020-10-15T16:36:28.258 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-10-15T16:36:28.378 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-10-15T16:36:28.656 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package jq. 2020-10-15T16:36:28.671 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-10-15T16:36:28.714 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-10-15T16:36:29.002 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-10-15T16:36:29.008 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package socat. 2020-10-15T16:36:29.016 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-10-15T16:36:29.025 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-10-15T16:36:29.060 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-10-15T16:36:29.066 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-10-15T16:36:29.437 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-10-15T16:36:29.451 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-10-15T16:36:29.495 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-15T16:36:29.548 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-joblib. 2020-10-15T16:36:29.562 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-10-15T16:36:29.606 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-joblib (0.11-1) ... 2020-10-15T16:36:29.773 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package xmlstarlet. 2020-10-15T16:36:29.787 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-10-15T16:36:29.831 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-10-15T16:36:29.959 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sklearn. 2020-10-15T16:36:30.064 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-10-15T16:36:30.064 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-10-15T16:36:30.193 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package ceph-test. 2020-10-15T16:36:30.210 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../085-ceph-test_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:30.252 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking ceph-test (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:30.541 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-10-15T16:36:30.556 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:30.599 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:31.728 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-monotonic. 2020-10-15T16:36:31.745 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-10-15T16:36:31.786 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-10-15T16:36:32.072 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-fasteners. 2020-10-15T16:36:32.087 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-10-15T16:36:32.131 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-10-15T16:36:32.450 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-httplib2. 2020-10-15T16:36:32.464 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-10-15T16:36:32.509 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-15T16:36:32.846 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyasn1. 2020-10-15T16:36:32.860 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-10-15T16:36:32.904 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-10-15T16:36:33.224 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-10-15T16:36:33.240 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-10-15T16:36:33.282 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-15T16:36:33.610 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rsa. 2020-10-15T16:36:33.624 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-10-15T16:36:33.677 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-10-15T16:36:33.914 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package libcephfs-dev. 2020-10-15T16:36:33.930 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:33.972 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking libcephfs-dev (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:34.014 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-10-15T16:36:34.028 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-10-15T16:36:34.073 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-15T16:36:34.241 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package nvme-cli. 2020-10-15T16:36:34.256 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-10-15T16:36:34.308 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-10-15T16:36:34.527 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-oauth2client. 2020-10-15T16:36:34.541 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-10-15T16:36:34.585 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-10-15T16:36:34.663 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-attr. 2020-10-15T16:36:34.672 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-10-15T16:36:34.713 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-attr (17.4.0-2) ... 2020-10-15T16:36:34.930 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-setuptools. 2020-10-15T16:36:34.944 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-10-15T16:36:34.988 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-10-15T16:36:35.016 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-funcsigs. 2020-10-15T16:36:35.031 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-10-15T16:36:35.074 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-10-15T16:36:35.360 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-pluggy. 2020-10-15T16:36:35.377 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-10-15T16:36:35.391 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-websocket. 2020-10-15T16:36:35.408 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-10-15T16:36:35.439 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-10-15T16:36:35.450 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-10-15T16:36:35.755 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-py. 2020-10-15T16:36:35.761 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-kubernetes. 2020-10-15T16:36:35.770 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-10-15T16:36:35.776 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-10-15T16:36:35.813 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-py (1.5.2-1) ... 2020-10-15T16:36:35.819 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-15T16:36:36.132 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python-pytest. 2020-10-15T16:36:36.148 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-10-15T16:36:36.191 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-10-15T16:36:36.249 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-10-15T16:36:36.263 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:36.316 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:36.519 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-attr. 2020-10-15T16:36:36.535 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-10-15T16:36:36.585 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-10-15T16:36:36.652 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-json-pointer. 2020-10-15T16:36:36.667 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-10-15T16:36:36.710 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-10-15T16:36:36.888 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-cycler. 2020-10-15T16:36:36.905 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-10-15T16:36:36.946 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-10-15T16:36:37.088 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jsonpatch. 2020-10-15T16:36:37.104 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-10-15T16:36:37.150 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-15T16:36:37.274 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-webencodings. 2020-10-15T16:36:37.288 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-10-15T16:36:37.333 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-10-15T16:36:37.450 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-10-15T16:36:37.464 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-6334-g417a7e57-1bionic_all.deb ... 2020-10-15T16:36:37.517 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-rook (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:37.643 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-html5lib. 2020-10-15T16:36:37.660 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-10-15T16:36:37.710 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-10-15T16:36:37.844 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libonig4:amd64. 2020-10-15T16:36:37.862 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-10-15T16:36:37.903 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-10-15T16:36:38.072 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-10-15T16:36:38.086 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-10-15T16:36:38.130 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-15T16:36:38.265 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libjq1:amd64. 2020-10-15T16:36:38.280 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-10-15T16:36:38.426 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-10-15T16:36:38.583 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyparsing. 2020-10-15T16:36:38.600 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-10-15T16:36:38.641 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-15T16:36:38.703 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package jq. 2020-10-15T16:36:38.719 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-10-15T16:36:38.762 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-10-15T16:36:38.994 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-tz. 2020-10-15T16:36:39.008 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-10-15T16:36:39.031 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package socat. 2020-10-15T16:36:39.047 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-10-15T16:36:39.052 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-tz (2018.3-2) ... 2020-10-15T16:36:39.089 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-10-15T16:36:39.355 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-matplotlib. 2020-10-15T16:36:39.371 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-10-15T16:36:39.422 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-15T16:36:39.477 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-10-15T16:36:39.491 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-10-15T16:36:39.535 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-15T16:36:39.846 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package xmlstarlet. 2020-10-15T16:36:39.863 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-10-15T16:36:39.904 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-10-15T16:36:40.136 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-olefile. 2020-10-15T16:36:40.151 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-10-15T16:36:40.194 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-10-15T16:36:40.234 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-test. 2020-10-15T16:36:40.249 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../085-ceph-test_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:40.292 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-test (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:40.513 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pastescript. 2020-10-15T16:36:40.529 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-10-15T16:36:40.572 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-10-15T16:36:40.992 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pil:amd64. 2020-10-15T16:36:41.007 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-10-15T16:36:41.050 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-15T16:36:41.437 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pluggy. 2020-10-15T16:36:41.454 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-10-15T16:36:41.495 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-10-15T16:36:41.798 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-py. 2020-10-15T16:36:41.812 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-10-15T16:36:41.856 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-py (1.5.2-1) ... 2020-10-15T16:36:42.202 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pyinotify. 2020-10-15T16:36:42.216 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-10-15T16:36:42.260 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-10-15T16:36:42.663 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-pytest. 2020-10-15T16:36:42.679 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-10-15T16:36:42.721 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-10-15T16:36:43.050 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-simplejson. 2020-10-15T16:36:43.064 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-10-15T16:36:43.108 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-10-15T16:36:43.528 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package python3-tk:amd64. 2020-10-15T16:36:43.541 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-10-15T16:36:43.586 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-15T16:36:43.988 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libcephfs-dev. 2020-10-15T16:36:44.003 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:44.046 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libcephfs-dev (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:44.048 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package radosgw. 2020-10-15T16:36:44.063 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../110-radosgw_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:44.106 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking radosgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:44.308 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package nvme-cli. 2020-10-15T16:36:44.323 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-10-15T16:36:44.366 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-10-15T16:36:44.730 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-attr. 2020-10-15T16:36:44.744 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-10-15T16:36:44.788 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-attr (17.4.0-2) ... 2020-10-15T16:36:45.093 INFO:teuthology.orchestra.run.smithi183.stdout:Selecting previously unselected package rbd-fuse. 2020-10-15T16:36:45.110 INFO:teuthology.orchestra.run.smithi183.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:45.142 INFO:teuthology.orchestra.run.smithi183.stdout:Unpacking rbd-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:45.166 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-funcsigs. 2020-10-15T16:36:45.180 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-10-15T16:36:45.224 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-10-15T16:36:45.471 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-10-15T16:36:45.527 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pluggy. 2020-10-15T16:36:45.543 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-10-15T16:36:45.586 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-10-15T16:36:45.772 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-15T16:36:45.911 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-py. 2020-10-15T16:36:45.929 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-10-15T16:36:45.972 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-py (1.5.2-1) ... 2020-10-15T16:36:46.149 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-15T16:36:46.283 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-15T16:36:46.283 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pytest. 2020-10-15T16:36:46.301 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-10-15T16:36:46.342 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-10-15T16:36:46.687 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-attr. 2020-10-15T16:36:46.701 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-10-15T16:36:46.745 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-10-15T16:36:46.913 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-15T16:36:47.032 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cycler. 2020-10-15T16:36:47.049 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-10-15T16:36:47.090 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-10-15T16:36:47.289 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-10-15T16:36:47.343 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webencodings. 2020-10-15T16:36:47.360 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-10-15T16:36:47.413 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-10-15T16:36:47.582 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-webencodings (0.5-2) ... 2020-10-15T16:36:47.662 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-html5lib. 2020-10-15T16:36:47.677 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-10-15T16:36:47.720 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-10-15T16:36:47.891 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-attr (17.4.0-2) ... 2020-10-15T16:36:48.057 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-10-15T16:36:48.072 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-10-15T16:36:48.092 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-10-15T16:36:48.115 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-15T16:36:48.210 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-py (1.5.2-1) ... 2020-10-15T16:36:48.545 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-10-15T16:36:48.577 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyparsing. 2020-10-15T16:36:48.594 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-10-15T16:36:48.644 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-15T16:36:48.751 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-10-15T16:36:48.980 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tz. 2020-10-15T16:36:48.995 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-10-15T16:36:49.038 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tz (2018.3-2) ... 2020-10-15T16:36:49.081 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-10-15T16:36:49.341 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-matplotlib. 2020-10-15T16:36:49.356 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-10-15T16:36:49.382 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-ceph-argparse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:49.399 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-15T16:36:49.674 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-10-15T16:36:49.950 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-10-15T16:36:50.051 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-15T16:36:50.082 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-olefile. 2020-10-15T16:36:50.097 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-10-15T16:36:50.140 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-10-15T16:36:50.501 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pastescript. 2020-10-15T16:36:50.516 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-10-15T16:36:50.568 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-10-15T16:36:50.939 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pil:amd64. 2020-10-15T16:36:50.955 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-10-15T16:36:50.963 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-10-15T16:36:50.997 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-15T16:36:51.272 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-10-15T16:36:51.359 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pluggy. 2020-10-15T16:36:51.374 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-10-15T16:36:51.417 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-10-15T16:36:51.565 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-py (1.5.2-1) ... 2020-10-15T16:36:51.703 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-py. 2020-10-15T16:36:51.720 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-10-15T16:36:51.761 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-py (1.5.2-1) ... 2020-10-15T16:36:51.791 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-10-15T16:36:52.100 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-10-15T16:36:52.157 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyinotify. 2020-10-15T16:36:52.171 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-10-15T16:36:52.173 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-10-15T16:36:52.215 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-10-15T16:36:52.402 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-10-15T16:36:52.526 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pytest. 2020-10-15T16:36:52.528 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-10-15T16:36:52.541 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-10-15T16:36:52.584 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-10-15T16:36:52.721 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-10-15T16:36:52.913 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-simplejson. 2020-10-15T16:36:52.927 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-10-15T16:36:52.971 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-10-15T16:36:53.022 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-15T16:36:53.291 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tk:amd64. 2020-10-15T16:36:53.307 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-10-15T16:36:53.349 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-15T16:36:53.356 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-10-15T16:36:53.657 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-10-15T16:36:53.803 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package radosgw. 2020-10-15T16:36:53.820 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../110-radosgw_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:53.861 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking radosgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:54.025 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-10-15T16:36:54.326 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-10-15T16:36:54.611 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:54.865 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package rbd-fuse. 2020-10-15T16:36:54.879 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-6334-g417a7e57-1bionic_amd64.deb ... 2020-10-15T16:36:54.923 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking rbd-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:55.238 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-10-15T16:36:55.269 INFO:teuthology.orchestra.run.smithi183.stdout:Adding system user cephadm....done 2020-10-15T16:36:55.458 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-10-15T16:36:55.545 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-15T16:36:55.743 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-15T16:36:55.852 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-10-15T16:36:55.921 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-15T16:36:55.961 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-10-15T16:36:56.064 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-15T16:36:56.287 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-10-15T16:36:56.589 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-10-15T16:36:56.667 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-15T16:36:56.714 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-10-15T16:36:57.049 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:57.051 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-10-15T16:36:57.345 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webencodings (0.5-2) ... 2020-10-15T16:36:57.350 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-monotonic (1.1-2) ... 2020-10-15T16:36:57.618 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-15T16:36:57.654 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-attr (17.4.0-2) ... 2020-10-15T16:36:57.727 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-15T16:36:57.874 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-10-15T16:36:58.017 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-py (1.5.2-1) ... 2020-10-15T16:36:58.345 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-10-15T16:36:58.352 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-10-15T16:36:58.542 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-10-15T16:36:58.688 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-15T16:36:58.777 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-10-15T16:36:58.818 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-10-15T16:36:58.846 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-10-15T16:36:59.056 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-10-15T16:36:59.173 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-ceph-argparse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:36:59.349 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-15T16:36:59.499 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-10-15T16:36:59.650 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-15T16:36:59.808 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-10-15T16:36:59.934 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-15T16:36:59.976 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-attr (17.4.0-2) ... 2020-10-15T16:37:00.252 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-10-15T16:37:00.445 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-15T16:37:00.872 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-10-15T16:37:00.896 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-10-15T16:37:00.961 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-10-15T16:37:01.223 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up liboath0 (2.6.1-1) ... 2020-10-15T16:37:01.255 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-10-15T16:37:01.383 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-15T16:37:01.492 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-15T16:37:01.548 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-py (1.5.2-1) ... 2020-10-15T16:37:01.727 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-10-15T16:37:01.775 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-10-15T16:37:01.828 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-10-15T16:37:02.109 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-10-15T16:37:02.199 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-10-15T16:37:02.204 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-15T16:37:02.330 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-10-15T16:37:02.436 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-10-15T16:37:02.587 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-10-15T16:37:02.622 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-tz (2018.3-2) ... 2020-10-15T16:37:02.797 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-10-15T16:37:02.932 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-10-15T16:37:03.089 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-15T16:37:03.216 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-10-15T16:37:03.416 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-10-15T16:37:03.526 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-joblib (0.11-1) ... 2020-10-15T16:37:03.709 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-10-15T16:37:03.877 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-routes (2.4.1-1) ... 2020-10-15T16:37:04.085 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-10-15T16:37:04.176 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-10-15T16:37:04.428 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-10-15T16:37:04.487 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-10-15T16:37:04.729 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:04.805 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-15T16:37:05.166 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-10-15T16:37:05.414 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user cephadm....done 2020-10-15T16:37:05.467 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up jq (1.5+dfsg-2) ... 2020-10-15T16:37:05.576 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-10-15T16:37:05.586 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-10-15T16:37:05.896 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-15T16:37:06.030 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-10-15T16:37:06.181 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-10-15T16:37:06.533 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-10-15T16:37:06.706 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-15T16:37:06.834 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-10-15T16:37:06.960 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-10-15T16:37:07.016 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-10-15T16:37:07.320 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:07.350 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-10-15T16:37:07.484 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-10-15T16:37:07.638 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-monotonic (1.1-2) ... 2020-10-15T16:37:07.827 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python-pytest (3.3.2-2) ... 2020-10-15T16:37:07.930 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-15T16:37:08.053 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-10-15T16:37:08.073 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-15T16:37:08.179 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-10-15T16:37:08.313 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librados2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:08.416 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-10-15T16:37:08.439 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-10-15T16:37:08.757 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libcephfs2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:08.801 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-15T16:37:08.883 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libcephfs-dev (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:08.891 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-10-15T16:37:08.923 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-10-15T16:37:09.000 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rados (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:09.102 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-10-15T16:37:09.153 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-10-15T16:37:09.394 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-10-15T16:37:09.463 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-15T16:37:09.670 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-10-15T16:37:09.780 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-15T16:37:10.005 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-15T16:37:10.140 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-attr (17.4.0-2) ... 2020-10-15T16:37:10.450 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-10-15T16:37:10.643 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-15T16:37:10.917 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-10-15T16:37:11.061 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-10-15T16:37:11.151 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-10-15T16:37:11.421 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liboath0 (2.6.1-1) ... 2020-10-15T16:37:11.622 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-15T16:37:11.738 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-10-15T16:37:11.765 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-15T16:37:12.031 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up libradosstriper1 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:12.033 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-10-15T16:37:12.157 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-cephfs (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:12.159 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-10-15T16:37:12.458 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librgw2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:12.561 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-15T16:37:12.575 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:12.687 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-10-15T16:37:12.778 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-15T16:37:12.778 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-15T16:37:13.021 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tz (2018.3-2) ... 2020-10-15T16:37:13.027 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up librbd1 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:13.128 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-10-15T16:37:13.218 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-10-15T16:37:13.250 INFO:teuthology.orchestra.run.smithi183.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-10-15T16:37:13.339 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-10-15T16:37:13.556 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-15T16:37:13.640 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-10-15T16:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-joblib (0.11-1) ... 2020-10-15T16:37:14.217 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-modules-core (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:14.310 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-routes (2.4.1-1) ... 2020-10-15T16:37:14.326 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-10-15T16:37:14.611 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:14.619 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-10-15T16:37:14.712 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up rbd-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:14.837 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up python3-rbd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:14.921 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-10-15T16:37:14.963 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:15.264 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-15T16:37:15.497 INFO:teuthology.orchestra.run.smithi183.stdout:Adding group ceph....done 2020-10-15T16:37:15.649 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-10-15T16:37:15.858 INFO:teuthology.orchestra.run.smithi183.stdout:Adding system user ceph....done 2020-10-15T16:37:15.950 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up jq (1.5+dfsg-2) ... 2020-10-15T16:37:16.050 INFO:teuthology.orchestra.run.smithi183.stdout:Setting system user ceph properties....done 2020-10-15T16:37:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-10-15T16:37:16.205 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-10-15T16:37:16.413 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-10-15T16:37:16.681 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up radosgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:16.950 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-15T16:37:16.950 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-15T16:37:17.282 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-15T16:37:17.357 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-test (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:17.483 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-base (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:17.802 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-10-15T16:37:17.883 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-10-15T16:37:18.052 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:18.218 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-10-15T16:37:18.264 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-15T16:37:18.265 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-15T16:37:18.336 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-10-15T16:37:18.637 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:18.704 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pytest (3.3.2-2) ... 2020-10-15T16:37:18.780 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-osd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:18.938 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-10-15T16:37:19.047 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-10-15T16:37:19.207 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librados2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.226 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-15T16:37:19.226 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-15T16:37:19.333 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-10-15T16:37:19.491 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.633 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-rook (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.659 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libcephfs2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.776 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mds (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.785 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libcephfs-dev (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.903 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rados (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:19.997 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-15T16:37:19.998 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-15T16:37:20.004 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-10-15T16:37:20.269 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mon (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:20.296 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-10-15T16:37:20.470 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-15T16:37:20.471 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-15T16:37:20.581 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-10-15T16:37:20.728 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-k8sevents (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:20.846 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph-mgr-dashboard (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:20.958 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-15T16:37:20.988 INFO:teuthology.orchestra.run.smithi183.stdout:Setting up ceph (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:21.115 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-10-15T16:37:21.308 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-10-15T16:37:21.904 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-10-15T16:37:22.734 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-10-15T16:37:23.035 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libradosstriper1 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:23.161 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cephfs (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:23.368 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-10-15T16:37:23.404 INFO:teuthology.orchestra.run.smithi183.stdout:ureadahead will be reprofiled on next reboot 2020-10-15T16:37:23.452 INFO:teuthology.orchestra.run.smithi183.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-10-15T16:37:23.462 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librgw2 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:23.563 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:23.795 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-15T16:37:23.795 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-15T16:37:23.868 INFO:teuthology.orchestra.run.smithi183.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-10-15T16:37:23.870 DEBUG:teuthology.parallel:result is None 2020-10-15T16:37:24.100 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librbd1 (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:24.232 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-10-15T16:37:24.322 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-10-15T16:37:24.354 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-10-15T16:37:24.644 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-15T16:37:25.338 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-modules-core (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:25.439 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-10-15T16:37:25.766 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:25.891 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up rbd-fuse (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:26.017 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rbd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:26.143 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-common (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:26.602 INFO:teuthology.orchestra.run.smithi038.stdout:Adding group ceph....done 2020-10-15T16:37:26.989 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user ceph....done 2020-10-15T16:37:27.190 INFO:teuthology.orchestra.run.smithi038.stdout:Setting system user ceph properties....done 2020-10-15T16:37:27.353 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-10-15T16:37:27.577 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-10-15T16:37:27.863 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up radosgw (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:28.113 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-15T16:37:28.113 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-15T16:37:28.523 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-test (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:28.649 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-base (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:28.995 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-10-15T16:37:29.301 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:29.520 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-15T16:37:29.520 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-15T16:37:29.827 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:29.970 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-osd (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:30.405 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-15T16:37:30.405 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-15T16:37:30.681 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-cephadm (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:30.823 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-rook (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:30.975 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mds (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:31.204 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-15T16:37:31.204 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-15T16:37:31.501 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mon (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:31.716 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-15T16:37:31.717 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-15T16:37:32.019 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-k8sevents (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:32.145 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-dashboard (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:32.279 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph (16.0.0-6334-g417a7e57-1bionic) ... 2020-10-15T16:37:32.381 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-10-15T16:37:32.565 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-10-15T16:37:34.661 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-10-15T16:37:34.708 INFO:teuthology.orchestra.run.smithi038.stdout:ureadahead will be reprofiled on next reboot 2020-10-15T16:37:34.762 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-10-15T16:37:35.210 INFO:teuthology.orchestra.run.smithi038.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-10-15T16:37:35.212 DEBUG:teuthology.parallel:result is None 2020-10-15T16:37:35.213 INFO:teuthology.orchestra.run.smithi183:> sudo lsb_release -is 2020-10-15T16:37:35.270 INFO:teuthology.orchestra.run.smithi183.stdout:Ubuntu 2020-10-15T16:37:35.275 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-15T16:37:35.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 2020-10-15T16:37:35.441 INFO:teuthology.orchestra.run.smithi183:> dpkg-query -W -f '${Version}' ceph 2020-10-15T16:37:35.465 INFO:teuthology.orchestra.run.smithi183.stdout:16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:37:35.466 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:37:35.467 INFO:teuthology.task.install:The correct ceph version 16.0.0-6334-g417a7e57-1bionic is installed. 2020-10-15T16:37:35.467 INFO:teuthology.orchestra.run.smithi038:> sudo lsb_release -is 2020-10-15T16:37:35.522 INFO:teuthology.orchestra.run.smithi038.stdout:Ubuntu 2020-10-15T16:37:35.528 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-15T16:37:35.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=417a7e57ff62f4d3de1c307caeb5ea5d04efadf1 2020-10-15T16:37:35.691 INFO:teuthology.orchestra.run.smithi038:> dpkg-query -W -f '${Version}' ceph 2020-10-15T16:37:35.715 INFO:teuthology.orchestra.run.smithi038.stdout:16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:37:35.716 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-6334-g417a7e57-1bionic 2020-10-15T16:37:35.716 INFO:teuthology.task.install:The correct ceph version 16.0.0-6334-g417a7e57-1bionic is installed. 2020-10-15T16:37:35.716 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-10-15T16:37:35.717 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:35.717 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-15T16:37:35.722 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-15T16:37:35.731 INFO:teuthology.orchestra.run.smithi183.stderr:24+1 records in 2020-10-15T16:37:35.732 INFO:teuthology.orchestra.run.smithi183.stderr:24+1 records out 2020-10-15T16:37:35.732 INFO:teuthology.orchestra.run.smithi183.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000219377 s, 57.7 MB/s 2020-10-15T16:37:35.733 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:37:35.733 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-15T16:37:35.761 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-15T16:37:35.771 INFO:teuthology.orchestra.run.smithi038.stderr:24+1 records in 2020-10-15T16:37:35.771 INFO:teuthology.orchestra.run.smithi038.stderr:24+1 records out 2020-10-15T16:37:35.772 INFO:teuthology.orchestra.run.smithi038.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000222474 s, 56.9 MB/s 2020-10-15T16:37:35.773 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-10-15T16:37:35.773 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:35.773 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2020-10-15T16:37:35.780 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-10-15T16:37:35.789 INFO:teuthology.orchestra.run.smithi183.stderr:5+1 records in 2020-10-15T16:37:35.790 INFO:teuthology.orchestra.run.smithi183.stderr:5+1 records out 2020-10-15T16:37:35.790 INFO:teuthology.orchestra.run.smithi183.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.724e-05 s, 31.4 MB/s 2020-10-15T16:37:35.791 INFO:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-10-15T16:37:35.846 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:37:35.847 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2020-10-15T16:37:35.852 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-10-15T16:37:35.860 INFO:teuthology.orchestra.run.smithi038.stderr:5+1 records in 2020-10-15T16:37:35.861 INFO:teuthology.orchestra.run.smithi038.stderr:5+1 records out 2020-10-15T16:37:35.861 INFO:teuthology.orchestra.run.smithi038.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.4344e-05 s, 32.4 MB/s 2020-10-15T16:37:35.862 INFO:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-10-15T16:37:35.923 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-10-15T16:37:35.924 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:35.924 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2020-10-15T16:37:35.929 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-10-15T16:37:35.938 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:35.939 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:35.939 INFO:teuthology.orchestra.run.smithi183.stderr:299 bytes copied, 7.021e-05 s, 4.3 MB/s 2020-10-15T16:37:35.940 INFO:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-10-15T16:37:35.998 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:37:35.999 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2020-10-15T16:37:36.003 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-10-15T16:37:36.012 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:37:36.012 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:37:36.012 INFO:teuthology.orchestra.run.smithi038.stderr:299 bytes copied, 6.6713e-05 s, 4.5 MB/s 2020-10-15T16:37:36.014 INFO:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-10-15T16:37:36.075 INFO:teuthology.run_tasks:Running task ceph... 2020-10-15T16:37:36.137 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-10-15T16:37:36.138 INFO:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2020-10-15T16:37:36.140 INFO:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2020-10-15T16:37:36.153 INFO:tasks.ceph:Disabling ceph logrotate... 2020-10-15T16:37:36.153 INFO:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2020-10-15T16:37:36.203 INFO:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2020-10-15T16:37:36.215 INFO:tasks.ceph:Creating extra log directories... 2020-10-15T16:37:36.216 INFO:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-10-15T16:37:36.259 INFO:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-10-15T16:37:36.282 INFO:tasks.ceph:Creating ceph cluster c1... 2020-10-15T16:37:36.282 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-10-15T16:37:36.283 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098', 'branch': 'wip-yuval-cls-fifo', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5527098', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuval-cls-fifo'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuval-cls-fifo', 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuval-cls-fifo', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.15069', 'targets': {'smithi183.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyyo5jDAgNCY2MAVJg7aLSIIrwNjaSiAqJyM9dxk4rMvo/VKE1meyht5fMhq+77fogrBP3Y8nfI9WRD1WvifWzXuLSftj7hTCcPEHJDRTUwBNeyg2FHMne2CrEAERvBNvkJkIRp79EeozjMGpNZ4Dxz8cG0kGiaUcCaWSprB3pLuCtg0oHKJ9M/C98g4fI4uXWmyfQNx5XgJJSJ5EOJcNw5jWS1xoq61U8APotg6FyxMSTh/YWpDb7wf1GHgxNFvQS+MhgWvZxR25aJ4wYev6HPy6neeVTlnbJH1SqCn7ChVswNcyVDMR6a5OTd4QMDLs+TIk7cnNN0TLF0mvlUCr3', 'smithi038.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1tZHJ9OhSMJ524FtRmYYH9mKdurGuoC1xzwIc+OnxZ3GQCn/VLl87OsLJ5F7GkCFOSabal/LT3fLJ+GztdJk8iEFZjn/LVrGWshqgyATInaHRf4KJdVRNBS2Yhvg9916OMVHQYHUbUPFwKjfsqCVQmEBc0dAYvsJH8IqpR7t0ByfgsY0A5MyGUdo/qHiq7j3yYU9yIr4m+mPiK6x2d3RGgkR6OO9zU86kRU6R8zBN7Yne6UNBnR3WWsEivDppY1cAb9Ty1XhLMRpiEd/cGxVRidv+RY8cXCFzf9hg6YPnkcSnszdXVEYXod/2Oau63yeM5L70mTdah/Gmr4E4qRP1'}} 2020-10-15T16:37:36.283 INFO:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-10-15T16:37:36.318 INFO:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-10-15T16:37:36.325 INFO:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2020-10-15T16:37:36.366 INFO:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2020-10-15T16:37:36.382 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:36.383 INFO:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2020-10-15T16:37:36.428 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-10-15T16:37:36.430 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_1 2020-10-15T16:37:36.430 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_2 2020-10-15T16:37:36.430 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_3 2020-10-15T16:37:36.430 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_4 2020-10-15T16:37:36.430 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:36.430 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:36.431 INFO:teuthology.orchestra.run.smithi183.stderr:72 bytes copied, 0.000164657 s, 437 kB/s 2020-10-15T16:37:36.431 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-10-15T16:37:36.431 INFO:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-10-15T16:37:36.480 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-10-15T16:37:36.480 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:37:36.480 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-10-15T16:37:36.480 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:37:36.480 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2020-10-15 16:34:59.883595975 +0000 2020-10-15T16:37:36.481 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2020-10-15 16:34:59.787643977 +0000 2020-10-15T16:37:36.481 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2020-10-15 16:34:59.787643977 +0000 2020-10-15T16:37:36.481 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2020-10-15T16:37:36.490 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2020-10-15T16:37:36.490 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2020-10-15T16:37:36.491 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000357779 s, 1.4 MB/s 2020-10-15T16:37:36.495 INFO:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-10-15T16:37:36.557 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-10-15T16:37:36.558 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:37:36.559 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-10-15T16:37:36.560 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:37:36.560 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2020-10-15 16:35:00.091491970 +0000 2020-10-15T16:37:36.561 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2020-10-15 16:34:59.991541972 +0000 2020-10-15T16:37:36.561 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2020-10-15 16:34:59.991541972 +0000 2020-10-15T16:37:36.562 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2020-10-15T16:37:36.567 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2020-10-15T16:37:36.568 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2020-10-15T16:37:36.568 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000366374 s, 1.4 MB/s 2020-10-15T16:37:36.571 INFO:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-10-15T16:37:36.639 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-10-15T16:37:36.640 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:37:36.640 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-10-15T16:37:36.640 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:37:36.641 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2020-10-15 16:35:00.303385965 +0000 2020-10-15T16:37:36.641 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2020-10-15 16:35:00.203435968 +0000 2020-10-15T16:37:36.641 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2020-10-15 16:35:00.203435968 +0000 2020-10-15T16:37:36.641 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2020-10-15T16:37:36.649 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2020-10-15T16:37:36.650 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2020-10-15T16:37:36.650 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000392684 s, 1.3 MB/s 2020-10-15T16:37:36.654 INFO:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-10-15T16:37:36.708 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-10-15T16:37:36.709 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:37:36.709 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-10-15T16:37:36.710 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:37:36.710 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2020-10-15 16:35:00.523275960 +0000 2020-10-15T16:37:36.711 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2020-10-15 16:35:00.435319962 +0000 2020-10-15T16:37:36.711 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2020-10-15 16:35:00.435319962 +0000 2020-10-15T16:37:36.712 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2020-10-15T16:37:36.717 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2020-10-15T16:37:36.717 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2020-10-15T16:37:36.717 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000372758 s, 1.4 MB/s 2020-10-15T16:37:36.723 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:37:36.724 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-15T16:37:36.725 INFO:tasks.ceph:Generating config... 2020-10-15T16:37:36.728 INFO:tasks.ceph:[client] debug rgw = 20 2020-10-15T16:37:36.729 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-10-15T16:37:36.730 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-10-15T16:37:36.731 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-15T16:37:36.731 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-10-15T16:37:36.732 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-10-15T16:37:36.733 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-10-15T16:37:36.733 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-10-15T16:37:36.734 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-10-15T16:37:36.734 INFO:tasks.ceph:[client] setgroup = ceph 2020-10-15T16:37:36.735 INFO:tasks.ceph:[client] setuser = ceph 2020-10-15T16:37:36.735 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-10-15T16:37:36.735 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-10-15T16:37:36.735 INFO:tasks.ceph:[mgr] debug ms = 1 2020-10-15T16:37:36.735 INFO:tasks.ceph:[mon] debug mon = 20 2020-10-15T16:37:36.736 INFO:tasks.ceph:[mon] debug ms = 1 2020-10-15T16:37:36.736 INFO:tasks.ceph:[mon] debug paxos = 20 2020-10-15T16:37:36.736 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-10-15T16:37:36.736 INFO:tasks.ceph:[osd] debug filestore = 20 2020-10-15T16:37:36.737 INFO:tasks.ceph:[osd] debug journal = 20 2020-10-15T16:37:36.737 INFO:tasks.ceph:[osd] debug ms = 20 2020-10-15T16:37:36.737 INFO:tasks.ceph:[osd] debug osd = 25 2020-10-15T16:37:36.737 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-10-15T16:37:36.737 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-10-15T16:37:36.738 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-10-15T16:37:36.738 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-10-15T16:37:36.738 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-10-15T16:37:36.738 INFO:tasks.ceph:Setting up c1.mon.a... 2020-10-15T16:37:36.739 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-10-15T16:37:36.796 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/c1.keyring 2020-10-15T16:37:36.798 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-10-15T16:37:36.875 INFO:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-15T16:37:36.930 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.183')] 2020-10-15T16:37:36.931 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.183', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-10-15T16:37:36.933 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:36.933 INFO:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-15T16:37:36.976 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-15T16:37:36.978 INFO:teuthology.orchestra.run.smithi183.stderr:6+1 records in 2020-10-15T16:37:36.978 INFO:teuthology.orchestra.run.smithi183.stderr:6+1 records out 2020-10-15T16:37:36.978 INFO:teuthology.orchestra.run.smithi183.stderr:3101 bytes (3.1 kB, 3.0 KiB) copied, 0.000108397 s, 28.6 MB/s 2020-10-15T16:37:36.979 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.183 --print /home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:37.052 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:37:37.052 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:37:37.052 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: generated fsid 1ed47c08-0a60-4d7f-868b-22bdfe649c24 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:epoch 0 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:fsid 1ed47c08-0a60-4d7f-868b-22bdfe649c24 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:last_changed 2020-10-15T16:37:37.055045+0000 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:created 2020-10-15T16:37:37.055045+0000 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:min_mon_release 0 (unknown) 2020-10-15T16:37:37.053 INFO:teuthology.orchestra.run.smithi183.stdout:election_strategy: 1 2020-10-15T16:37:37.054 INFO:teuthology.orchestra.run.smithi183.stdout:0: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.a 2020-10-15T16:37:37.054 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-10-15T16:37:37.055 INFO:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-10-15T16:37:37.102 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 1ed47c08-0a60-4d7f-868b-22bdfe649c24... 2020-10-15T16:37:37.103 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-10-15T16:37:37.105 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-10-15T16:37:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2020-10-15T16:37:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2020-10-15T16:37:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-15T16:37:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2020-10-15T16:37:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2020-10-15T16:37:37.148 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.149 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2020-10-15T16:37:37.150 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2020-10-15T16:37:37.151 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.152 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = 172.21.15.183 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: osd heartbeat grace = 40 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 1ed47c08-0a60-4d7f-868b-22bdfe649c24 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2020-10-15T16:37:37.153 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.154 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.155 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2020-10-15T16:37:37.156 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: debug filestore = 20 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: debug journal = 20 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 20 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 25 2020-10-15T16:37:37.157 INFO:teuthology.orchestra.run.smithi038.stdout: osd fast shutdown = False 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: rocksdb delete range threshold = 0 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2020-10-15T16:37:37.158 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2020-10-15T16:37:37.159 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:37:37.160 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd crush smoke test = False 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-15T16:37:37.161 INFO:teuthology.orchestra.run.smithi038.stdout: debug rgw = 20 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt require ssl = False 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt s3 kms backend = testing 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw curl low speed time = 300 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw data log num shards = 4 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw md log max shards = 4 2020-10-15T16:37:37.162 INFO:teuthology.orchestra.run.smithi038.stdout: rgw sync log trim interval = 0 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout: rgw sync obj etag verify = True 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout: setgroup = ceph 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout: setuser = ceph 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_omap_entries_per_request = 10 2020-10-15T16:37:37.163 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2020-10-15T16:37:37.164 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_omap_entries_per_request = 1000 2020-10-15T16:37:37.164 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2020-10-15T16:37:37.164 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_omap_entries_per_request = 10000 2020-10-15T16:37:37.178 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2020-10-15T16:37:37.179 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2020-10-15T16:37:37.179 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-15T16:37:37.179 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2020-10-15T16:37:37.179 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.179 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2020-10-15T16:37:37.180 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.180 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2020-10-15T16:37:37.180 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.180 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2020-10-15T16:37:37.180 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2020-10-15T16:37:37.180 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.181 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2020-10-15T16:37:37.182 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-15T16:37:37.182 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.182 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2020-10-15T16:37:37.182 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.182 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2020-10-15T16:37:37.182 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2020-10-15T16:37:37.183 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2020-10-15T16:37:37.183 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2020-10-15T16:37:37.183 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2020-10-15T16:37:37.183 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2020-10-15T16:37:37.183 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2020-10-15T16:37:37.183 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-15T16:37:37.184 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-15T16:37:37.184 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2020-10-15T16:37:37.184 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.184 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-15T16:37:37.184 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.184 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = 172.21.15.183 2020-10-15T16:37:37.185 INFO:teuthology.orchestra.run.smithi183.stdout: osd heartbeat grace = 40 2020-10-15T16:37:37.186 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 1ed47c08-0a60-4d7f-868b-22bdfe649c24 2020-10-15T16:37:37.186 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.186 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2020-10-15T16:37:37.187 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2020-10-15T16:37:37.187 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.187 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2020-10-15T16:37:37.187 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2020-10-15T16:37:37.187 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.187 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2020-10-15T16:37:37.188 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2020-10-15T16:37:37.188 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.188 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2020-10-15T16:37:37.188 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2020-10-15T16:37:37.188 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2020-10-15T16:37:37.188 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.189 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2020-10-15T16:37:37.190 INFO:teuthology.orchestra.run.smithi183.stdout: debug filestore = 20 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout: debug journal = 20 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 20 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 25 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout: osd fast shutdown = False 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout: rocksdb delete range threshold = 0 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.191 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:37:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2020-10-15T16:37:37.193 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd crush smoke test = False 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:37.194 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: debug rgw = 20 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: rgw crypt require ssl = False 2020-10-15T16:37:37.195 INFO:teuthology.orchestra.run.smithi183.stdout: rgw crypt s3 kms backend = testing 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: rgw curl low speed time = 300 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: rgw data log num shards = 4 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: rgw md log max shards = 4 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: rgw sync log trim interval = 0 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: rgw sync obj etag verify = True 2020-10-15T16:37:37.196 INFO:teuthology.orchestra.run.smithi183.stdout: setgroup = ceph 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout: setuser = ceph 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout: osd_max_omap_entries_per_request = 10 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout: osd_max_omap_entries_per_request = 1000 2020-10-15T16:37:37.197 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2020-10-15T16:37:37.198 INFO:teuthology.orchestra.run.smithi183.stdout: osd_max_omap_entries_per_request = 10000 2020-10-15T16:37:37.198 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-10-15T16:37:37.199 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-10-15T16:37:37.285 INFO:tasks.ceph:Copying monmap to all nodes... 2020-10-15T16:37:37.286 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:37.286 INFO:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-10-15T16:37:37.291 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-10-15T16:37:37.292 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.] 2020-10-15T16:37:37.293 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDQeohf1JgmNBAASYP2ReAH8YyKmOtDK7sYNA== 2020-10-15T16:37:37.293 INFO:teuthology.orchestra.run.smithi183.stdout:[client.admin] 2020-10-15T16:37:37.293 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohfAV8CERAAqqzxKPv6N7hPiXXGtHe21g== 2020-10-15T16:37:37.293 INFO:teuthology.orchestra.run.smithi183.stdout: caps mds = "allow *" 2020-10-15T16:37:37.293 INFO:teuthology.orchestra.run.smithi183.stdout: caps mgr = "allow *" 2020-10-15T16:37:37.293 INFO:teuthology.orchestra.run.smithi183.stdout: caps mon = "allow *" 2020-10-15T16:37:37.294 INFO:teuthology.orchestra.run.smithi183.stdout: caps osd = "allow *" 2020-10-15T16:37:37.294 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:37.294 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:37.294 INFO:teuthology.orchestra.run.smithi183.stderr:206 bytes copied, 0.00019033 s, 1.1 MB/s 2020-10-15T16:37:37.294 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:37.295 INFO:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-10-15T16:37:37.340 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-10-15T16:37:37.342 INFO:teuthology.orchestra.run.smithi183.stdout: ��| 2020-10-15T16:37:37.342 INFO:teuthology.orchestra.run.smithi183.stdout:`M��"��d�$�z�_+�G�z�_+�G�aXa �����a 2020-10-15T16:37:37.342 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:37.342 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:37.342 INFO:teuthology.orchestra.run.smithi183.stderr:201 bytes copied, 0.000160047 s, 1.3 MB/s 2020-10-15T16:37:37.343 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:37:37.343 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:37.343 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/c1.keyring 2020-10-15T16:37:37.344 INFO:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-15T16:37:37.388 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-10-15T16:37:37.397 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:37.398 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:37.398 INFO:teuthology.orchestra.run.smithi183.stderr:206 bytes copied, 0.000118057 s, 1.7 MB/s 2020-10-15T16:37:37.398 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-15T16:37:37.407 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:37.408 INFO:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:37.452 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:37.454 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:37.454 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:37.454 INFO:teuthology.orchestra.run.smithi183.stderr:201 bytes copied, 0.000125517 s, 1.6 MB/s 2020-10-15T16:37:37.454 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:37:37.455 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:37:37.455 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/c1.keyring 2020-10-15T16:37:37.455 INFO:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-15T16:37:37.460 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-10-15T16:37:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:37:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:37:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:206 bytes copied, 0.000101097 s, 2.0 MB/s 2020-10-15T16:37:37.470 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-15T16:37:37.480 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:37:37.481 INFO:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:37.525 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:37.527 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:37:37.527 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:37:37.527 INFO:teuthology.orchestra.run.smithi038.stderr:201 bytes copied, 7.2386e-05 s, 2.8 MB/s 2020-10-15T16:37:37.527 INFO:tasks.ceph:Setting up mon nodes... 2020-10-15T16:37:37.527 INFO:tasks.ceph:Setting up mgr nodes... 2020-10-15T16:37:37.528 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-10-15T16:37:37.572 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-10-15T16:37:37.574 INFO:tasks.ceph:Setting up mds nodes... 2020-10-15T16:37:37.574 INFO:tasks.ceph_client:Setting up client nodes... 2020-10-15T16:37:37.575 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-10-15T16:37:37.648 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/c1.client.0.keyring 2020-10-15T16:37:37.660 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-10-15T16:37:37.735 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/c1.client.1.keyring 2020-10-15T16:37:37.746 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-10-15T16:37:37.747 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-15T16:37:37.748 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-10-15T16:37:37.803 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:37:37.803 INFO:tasks.ceph:role: c1.osd.0 2020-10-15T16:37:37.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:37:37.804 INFO:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-10-15T16:37:38.326 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-10-15T16:37:38.326 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-15T16:37:38.326 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-15T16:37:38.327 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-15T16:37:38.327 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2020-10-15T16:37:38.327 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-15T16:37:38.327 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-15T16:37:38.327 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-15T16:37:38.327 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-15T16:37:38.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2020-10-15T16:37:38.329 INFO:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-10-15T16:37:38.396 INFO:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-10-15T16:37:38.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2020-10-15T16:37:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-15T16:37:38.409 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-10-15T16:37:38.465 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:37:38.465 INFO:tasks.ceph:role: c1.osd.1 2020-10-15T16:37:38.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:37:38.466 INFO:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-10-15T16:37:38.979 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-10-15T16:37:38.979 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-15T16:37:38.980 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-15T16:37:38.980 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-15T16:37:38.980 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2020-10-15T16:37:38.980 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-15T16:37:38.980 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-15T16:37:38.980 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-15T16:37:38.981 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-15T16:37:38.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2020-10-15T16:37:38.982 INFO:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-10-15T16:37:39.054 INFO:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-10-15T16:37:39.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2020-10-15T16:37:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-15T16:37:39.066 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-10-15T16:37:39.122 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:37:39.123 INFO:tasks.ceph:role: c1.osd.2 2020-10-15T16:37:39.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:37:39.123 INFO:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-10-15T16:37:39.628 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-10-15T16:37:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-15T16:37:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-15T16:37:39.629 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-15T16:37:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2020-10-15T16:37:39.629 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-15T16:37:39.629 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-15T16:37:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-15T16:37:39.630 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-15T16:37:39.630 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2020-10-15T16:37:39.631 INFO:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-10-15T16:37:39.678 INFO:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-10-15T16:37:39.689 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: /sbin/restorecon: command not found 2020-10-15T16:37:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-15T16:37:39.691 INFO:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:39.774 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:39.774+0000 7f8818cc0d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-10-15T16:37:39.776 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:39.774+0000 7f8818cc0d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-10-15T16:37:39.776 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:39.774+0000 7f8818cc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-10-15T16:37:39.776 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:39.774+0000 7f8818cc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-10-15T16:37:39.776 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:39.774+0000 7f8818cc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-10-15T16:37:39.779 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:39.778+0000 7f8818cc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-10-15T16:37:41.952 INFO:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-10-15T16:37:42.015 INFO:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:42.096 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:42.094+0000 7f0b4a7cdd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-10-15T16:37:42.096 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:42.094+0000 7f0b4a7cdd80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-10-15T16:37:42.096 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:42.094+0000 7f0b4a7cdd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-10-15T16:37:42.097 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:42.094+0000 7f0b4a7cdd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-10-15T16:37:42.097 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:42.094+0000 7f0b4a7cdd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-10-15T16:37:42.097 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:42.098+0000 7f0b4a7cdd80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-10-15T16:37:44.284 INFO:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-10-15T16:37:44.344 INFO:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:44.426 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:44.426+0000 7f791953fd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-10-15T16:37:44.426 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:44.426+0000 7f791953fd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-10-15T16:37:44.426 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:44.426+0000 7f791953fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-10-15T16:37:44.427 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:44.426+0000 7f791953fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-10-15T16:37:44.427 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:44.426+0000 7f791953fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-10-15T16:37:44.427 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:37:44.426+0000 7f791953fd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-10-15T16:37:46.599 INFO:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-10-15T16:37:46.655 INFO:tasks.ceph:Reading keys from all nodes... 2020-10-15T16:37:46.656 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:46.656 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-10-15T16:37:46.701 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-10-15T16:37:46.709 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2020-10-15T16:37:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohfP2dBIhAAP+cj3GrLmUQlmrwmK/0yRw== 2020-10-15T16:37:46.710 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:46.710 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:46.710 INFO:teuthology.orchestra.run.smithi183.stderr:56 bytes copied, 0.000178174 s, 314 kB/s 2020-10-15T16:37:46.711 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:46.712 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-10-15T16:37:46.777 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-10-15T16:37:46.786 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2020-10-15T16:37:46.786 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDTeohfLsBNLhAAjNU8F/Ssxyerpe43Zgc+bg== 2020-10-15T16:37:46.787 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:46.788 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:46.788 INFO:teuthology.orchestra.run.smithi183.stderr:56 bytes copied, 0.000208661 s, 268 kB/s 2020-10-15T16:37:46.791 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:46.792 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-10-15T16:37:46.845 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-10-15T16:37:46.853 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2020-10-15T16:37:46.854 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDWeohf6q/gBRAAsO/Sjf1AdEPBgZ/ldXA6fg== 2020-10-15T16:37:46.854 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:46.855 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:46.855 INFO:teuthology.orchestra.run.smithi183.stderr:56 bytes copied, 0.000201494 s, 278 kB/s 2020-10-15T16:37:46.856 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:46.857 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-10-15T16:37:46.906 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-10-15T16:37:46.914 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2020-10-15T16:37:46.915 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDYeohfGqSIGRAAbuUkJfR7CwEKC/S4VNiqiQ== 2020-10-15T16:37:46.915 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:46.915 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:46.915 INFO:teuthology.orchestra.run.smithi183.stderr:56 bytes copied, 0.000207723 s, 270 kB/s 2020-10-15T16:37:46.916 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:46.917 INFO:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-10-15T16:37:46.964 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-10-15T16:37:46.965 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2020-10-15T16:37:46.965 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohfc7rRJhAA3XNOPFxiJGza1PyLm7NBFg== 2020-10-15T16:37:46.966 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:46.966 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:46.966 INFO:teuthology.orchestra.run.smithi183.stderr:59 bytes copied, 0.000156474 s, 377 kB/s 2020-10-15T16:37:46.966 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:46.967 INFO:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-10-15T16:37:47.011 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-10-15T16:37:47.013 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2020-10-15T16:37:47.013 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohf4Zv4KxAA/ZtYOmC+b1g5T8dBIcLUpA== 2020-10-15T16:37:47.013 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:47.013 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:47.013 INFO:teuthology.orchestra.run.smithi183.stderr:59 bytes copied, 0.000181435 s, 325 kB/s 2020-10-15T16:37:47.014 INFO:tasks.ceph:Adding keys to all mons... 2020-10-15T16:37:47.014 INFO:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/c1.keyring 2020-10-15T16:37:47.069 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2020-10-15T16:37:47.069 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohfP2dBIhAAP+cj3GrLmUQlmrwmK/0yRw== 2020-10-15T16:37:47.069 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2020-10-15T16:37:47.069 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDTeohfLsBNLhAAjNU8F/Ssxyerpe43Zgc+bg== 2020-10-15T16:37:47.069 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDWeohf6q/gBRAAsO/Sjf1AdEPBgZ/ldXA6fg== 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDYeohfGqSIGRAAbuUkJfR7CwEKC/S4VNiqiQ== 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohfc7rRJhAA3XNOPFxiJGza1PyLm7NBFg== 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2020-10-15T16:37:47.070 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDReohf4Zv4KxAA/ZtYOmC+b1g5T8dBIcLUpA== 2020-10-15T16:37:47.071 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-10-15T16:37:47.153 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-15T16:37:47.236 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-15T16:37:47.321 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-15T16:37:47.405 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-15T16:37:47.494 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-15T16:37:47.533 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-10-15T16:37:47.534 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-10-15T16:37:47.589 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-10-15T16:37:47.686 INFO:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-10-15T16:37:47.700 INFO:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-10-15T16:37:47.745 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-10-15T16:37:47.746 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-10-15T16:37:47.746 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-10-15T16:37:47.789 INFO:tasks.ceph.c1.mon.a:Started 2020-10-15T16:37:47.790 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-10-15T16:37:47.790 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-10-15T16:37:47.791 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-10-15T16:37:47.792 INFO:tasks.ceph.c1.mgr.x:Started 2020-10-15T16:37:47.793 INFO:tasks.ceph:Setting crush tunables to default 2020-10-15T16:37:47.793 INFO:teuthology.orchestra.run.smithi183:> sudo ceph --cluster c1 osd crush tunables default 2020-10-15T16:37:48.260 INFO:teuthology.orchestra.run.smithi183.stderr:adjusted tunables profile to default 2020-10-15T16:37:48.272 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-10-15T16:37:48.272 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:48.272 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-10-15T16:37:48.275 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-10-15T16:37:48.280 INFO:teuthology.orchestra.run.smithi183.stdout:bb9ac728-1df1-4164-a58e-dd20186a1168 2020-10-15T16:37:48.280 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:48.280 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:48.280 INFO:teuthology.orchestra.run.smithi183.stderr:37 bytes copied, 0.00020273 s, 183 kB/s 2020-10-15T16:37:48.281 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:48.281 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-10-15T16:37:48.328 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-10-15T16:37:48.336 INFO:teuthology.orchestra.run.smithi183.stdout:62b5f66a-b15d-4798-9a3b-a1c698cdf027 2020-10-15T16:37:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:37 bytes copied, 9.1868e-05 s, 403 kB/s 2020-10-15T16:37:48.338 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:37:48.338 INFO:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-10-15T16:37:48.384 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-10-15T16:37:48.392 INFO:teuthology.orchestra.run.smithi183.stdout:f3e73d60-7358-4c49-b957-d289197dac18 2020-10-15T16:37:48.392 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:37:48.392 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:37:48.393 INFO:teuthology.orchestra.run.smithi183.stderr:37 bytes copied, 0.000150883 s, 245 kB/s 2020-10-15T16:37:48.393 INFO:teuthology.orchestra.run.smithi183:> sudo ceph --cluster c1 osd new bb9ac728-1df1-4164-a58e-dd20186a1168 0 2020-10-15T16:37:48.654 INFO:teuthology.orchestra.run.smithi183.stdout:0 2020-10-15T16:37:48.670 INFO:teuthology.orchestra.run.smithi183:> sudo ceph --cluster c1 osd new 62b5f66a-b15d-4798-9a3b-a1c698cdf027 1 2020-10-15T16:37:48.879 INFO:teuthology.orchestra.run.smithi183.stdout:1 2020-10-15T16:37:48.891 INFO:teuthology.orchestra.run.smithi183:> sudo ceph --cluster c1 osd new f3e73d60-7358-4c49-b957-d289197dac18 2 2020-10-15T16:37:49.102 INFO:teuthology.orchestra.run.smithi183.stdout:2 2020-10-15T16:37:49.114 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-10-15T16:37:49.115 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-10-15T16:37:49.116 INFO:tasks.ceph.c1.osd.0:Started 2020-10-15T16:37:49.117 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-10-15T16:37:49.117 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-10-15T16:37:49.119 INFO:tasks.ceph.c1.osd.1:Started 2020-10-15T16:37:49.119 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-10-15T16:37:49.119 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-10-15T16:37:49.121 INFO:tasks.ceph.c1.osd.2:Started 2020-10-15T16:37:49.121 INFO:tasks.ceph:Waiting for OSDs to come up 2020-10-15T16:37:49.425 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-10-15T16:37:49.615 INFO:teuthology.misc.health.smithi183.stdout: 2020-10-15T16:37:49.616 INFO:teuthology.misc.health.smithi183.stdout:{"epoch":5,"fsid":"1ed47c08-0a60-4d7f-868b-22bdfe649c24","created":"2020-10-15T16:37:47.910220+0000","modified":"2020-10-15T16:37:49.102703+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"bb9ac728-1df1-4164-a58e-dd20186a1168","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"62b5f66a-b15d-4798-9a3b-a1c698cdf027","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3e73d60-7358-4c49-b957-d289197dac18","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:37:49.634 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-10-15T16:37:49.709 INFO:tasks.ceph.c1.osd.0.smithi183.stderr:2020-10-15T16:37:49.710+0000 7ff15ca3ed80 -1 Falling back to public interface 2020-10-15T16:37:49.729 INFO:tasks.ceph.c1.osd.1.smithi183.stderr:2020-10-15T16:37:49.730+0000 7f62fece3d80 -1 Falling back to public interface 2020-10-15T16:37:49.745 INFO:tasks.ceph.c1.osd.2.smithi183.stderr:2020-10-15T16:37:49.746+0000 7f12cce59d80 -1 Falling back to public interface 2020-10-15T16:37:50.307 INFO:tasks.ceph.c1.osd.0.smithi183.stderr:2020-10-15T16:37:50.306+0000 7ff15ca3ed80 -1 osd.0 0 log_to_monitors {default=true} 2020-10-15T16:37:50.378 INFO:tasks.ceph.c1.osd.2.smithi183.stderr:2020-10-15T16:37:50.378+0000 7f12cce59d80 -1 osd.2 0 log_to_monitors {default=true} 2020-10-15T16:37:50.382 INFO:tasks.ceph.c1.osd.1.smithi183.stderr:2020-10-15T16:37:50.382+0000 7f62fece3d80 -1 osd.1 0 log_to_monitors {default=true} 2020-10-15T16:37:51.942 INFO:tasks.ceph.c1.osd.2.smithi183.stderr:2020-10-15T16:37:51.942+0000 7f12c4324700 -1 osd.2 0 waiting for initial osdmap 2020-10-15T16:37:51.943 INFO:tasks.ceph.c1.osd.0.smithi183.stderr:2020-10-15T16:37:51.942+0000 7ff153f09700 -1 osd.0 0 waiting for initial osdmap 2020-10-15T16:37:51.943 INFO:tasks.ceph.c1.osd.1.smithi183.stderr:2020-10-15T16:37:51.942+0000 7f62f61ae700 -1 osd.1 0 waiting for initial osdmap 2020-10-15T16:37:55.937 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-10-15T16:37:56.219 INFO:teuthology.misc.health.smithi183.stdout: 2020-10-15T16:37:56.219 INFO:teuthology.misc.health.smithi183.stdout:{"epoch":10,"fsid":"1ed47c08-0a60-4d7f-868b-22bdfe649c24","created":"2020-10-15T16:37:47.910220+0000","modified":"2020-10-15T16:37:54.956010+0000","last_up_change":"2020-10-15T16:37:52.936722+0000","last_in_change":"2020-10-15T16:37:52.936722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:37:51.875490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"bb9ac728-1df1-4164-a58e-dd20186a1168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":13343},{"type":"v1","addr":"172.21.15.183:6801","nonce":13343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":13343},{"type":"v1","addr":"172.21.15.183:6803","nonce":13343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":13343},{"type":"v1","addr":"172.21.15.183:6807","nonce":13343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":13343},{"type":"v1","addr":"172.21.15.183:6805","nonce":13343}]},"public_addr":"172.21.15.183:6801/13343","cluster_addr":"172.21.15.183:6803/13343","heartbeat_back_addr":"172.21.15.183:6807/13343","heartbeat_front_addr":"172.21.15.183:6805/13343","state":["exists","up"]},{"osd":1,"uuid":"62b5f66a-b15d-4798-9a3b-a1c698cdf027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":13345},{"type":"v1","addr":"172.21.15.183:6809","nonce":13345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":13345},{"type":"v1","addr":"172.21.15.183:6811","nonce":13345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":13345},{"type":"v1","addr":"172.21.15.183:6815","nonce":13345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":13345},{"type":"v1","addr":"172.21.15.183:6813","nonce":13345}]},"public_addr":"172.21.15.183:6809/13345","cluster_addr":"172.21.15.183:6811/13345","heartbeat_back_addr":"172.21.15.183:6815/13345","heartbeat_front_addr":"172.21.15.183:6813/13345","state":["exists","up"]},{"osd":2,"uuid":"f3e73d60-7358-4c49-b957-d289197dac18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":13344},{"type":"v1","addr":"172.21.15.183:6817","nonce":13344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":13344},{"type":"v1","addr":"172.21.15.183:6819","nonce":13344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":13344},{"type":"v1","addr":"172.21.15.183:6823","nonce":13344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":13344},{"type":"v1","addr":"172.21.15.183:6821","nonce":13344}]},"public_addr":"172.21.15.183:6817/13344","cluster_addr":"172.21.15.183:6819/13344","heartbeat_back_addr":"172.21.15.183:6823/13344","heartbeat_front_addr":"172.21.15.183:6821/13344","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.292229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.372218+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.430720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:37:56.231 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-10-15T16:37:56.232 INFO:tasks.ceph:Creating RBD pool 2020-10-15T16:37:56.232 INFO:teuthology.orchestra.run.smithi183:> sudo ceph --cluster c1 osd pool create rbd 8 2020-10-15T16:37:56.986 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'rbd' created 2020-10-15T16:37:57.002 INFO:teuthology.orchestra.run.smithi183:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-10-15T16:37:57.996 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rbd' on pool 'rbd' 2020-10-15T16:37:58.012 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-10-15T16:37:58.013 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-10-15T16:37:58.013 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-10-15T16:37:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:58.296 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":12,"fsid":"1ed47c08-0a60-4d7f-868b-22bdfe649c24","created":"2020-10-15T16:37:47.910220+0000","modified":"2020-10-15T16:37:57.979009+0000","last_up_change":"2020-10-15T16:37:52.936722+0000","last_in_change":"2020-10-15T16:37:52.936722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:37:51.875490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-15T16:37:56.515913+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bb9ac728-1df1-4164-a58e-dd20186a1168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":13343},{"type":"v1","addr":"172.21.15.183:6801","nonce":13343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":13343},{"type":"v1","addr":"172.21.15.183:6803","nonce":13343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":13343},{"type":"v1","addr":"172.21.15.183:6807","nonce":13343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":13343},{"type":"v1","addr":"172.21.15.183:6805","nonce":13343}]},"public_addr":"172.21.15.183:6801/13343","cluster_addr":"172.21.15.183:6803/13343","heartbeat_back_addr":"172.21.15.183:6807/13343","heartbeat_front_addr":"172.21.15.183:6805/13343","state":["exists","up"]},{"osd":1,"uuid":"62b5f66a-b15d-4798-9a3b-a1c698cdf027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":13345},{"type":"v1","addr":"172.21.15.183:6809","nonce":13345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":13345},{"type":"v1","addr":"172.21.15.183:6811","nonce":13345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":13345},{"type":"v1","addr":"172.21.15.183:6815","nonce":13345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":13345},{"type":"v1","addr":"172.21.15.183:6813","nonce":13345}]},"public_addr":"172.21.15.183:6809/13345","cluster_addr":"172.21.15.183:6811/13345","heartbeat_back_addr":"172.21.15.183:6815/13345","heartbeat_front_addr":"172.21.15.183:6813/13345","state":["exists","up"]},{"osd":2,"uuid":"f3e73d60-7358-4c49-b957-d289197dac18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":13344},{"type":"v1","addr":"172.21.15.183:6817","nonce":13344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":13344},{"type":"v1","addr":"172.21.15.183:6819","nonce":13344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":13344},{"type":"v1","addr":"172.21.15.183:6823","nonce":13344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":13344},{"type":"v1","addr":"172.21.15.183:6821","nonce":13344}]},"public_addr":"172.21.15.183:6817/13344","cluster_addr":"172.21.15.183:6819/13344","heartbeat_back_addr":"172.21.15.183:6823/13344","heartbeat_front_addr":"172.21.15.183:6821/13344","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.292229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.372218+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.430720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:37:58.311 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-10-15T16:37:51.875490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-10-15T16:37:56.515913+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-10-15T16:37:58.311 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-10-15T16:37:58.597 INFO:teuthology.orchestra.run.smithi183.stdout:pg_num: 1 2020-10-15T16:37:58.612 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-10-15T16:37:58.908 INFO:teuthology.orchestra.run.smithi183.stdout:pg_num: 8 2020-10-15T16:37:58.922 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-10-15T16:37:58.923 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-10-15T16:37:58.923 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-10-15T16:37:59.250 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:59.264 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":13160},{"type":"v1","addr":"172.21.15.183:6825","nonce":13160}]},"active_addr":"172.21.15.183:6825/13160","active_change":"2020-10-15T16:37:51.845147+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":4101962476}]},{"addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":1000593808}]},{"addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3246761538}]}]}} 2020-10-15T16:37:59.275 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-10-15T16:37:59.276 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-10-15T16:37:59.276 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-10-15T16:37:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:59.563 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":13,"fsid":"1ed47c08-0a60-4d7f-868b-22bdfe649c24","created":"2020-10-15T16:37:47.910220+0000","modified":"2020-10-15T16:37:58.985599+0000","last_up_change":"2020-10-15T16:37:52.936722+0000","last_in_change":"2020-10-15T16:37:52.936722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:37:51.875490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-15T16:37:56.515913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bb9ac728-1df1-4164-a58e-dd20186a1168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":13343},{"type":"v1","addr":"172.21.15.183:6801","nonce":13343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":13343},{"type":"v1","addr":"172.21.15.183:6803","nonce":13343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":13343},{"type":"v1","addr":"172.21.15.183:6807","nonce":13343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":13343},{"type":"v1","addr":"172.21.15.183:6805","nonce":13343}]},"public_addr":"172.21.15.183:6801/13343","cluster_addr":"172.21.15.183:6803/13343","heartbeat_back_addr":"172.21.15.183:6807/13343","heartbeat_front_addr":"172.21.15.183:6805/13343","state":["exists","up"]},{"osd":1,"uuid":"62b5f66a-b15d-4798-9a3b-a1c698cdf027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":13345},{"type":"v1","addr":"172.21.15.183:6809","nonce":13345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":13345},{"type":"v1","addr":"172.21.15.183:6811","nonce":13345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":13345},{"type":"v1","addr":"172.21.15.183:6815","nonce":13345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":13345},{"type":"v1","addr":"172.21.15.183:6813","nonce":13345}]},"public_addr":"172.21.15.183:6809/13345","cluster_addr":"172.21.15.183:6811/13345","heartbeat_back_addr":"172.21.15.183:6815/13345","heartbeat_front_addr":"172.21.15.183:6813/13345","state":["exists","up"]},{"osd":2,"uuid":"f3e73d60-7358-4c49-b957-d289197dac18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":13344},{"type":"v1","addr":"172.21.15.183:6817","nonce":13344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":13344},{"type":"v1","addr":"172.21.15.183:6819","nonce":13344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":13344},{"type":"v1","addr":"172.21.15.183:6823","nonce":13344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":13344},{"type":"v1","addr":"172.21.15.183:6821","nonce":13344}]},"public_addr":"172.21.15.183:6817/13344","cluster_addr":"172.21.15.183:6819/13344","heartbeat_back_addr":"172.21.15.183:6823/13344","heartbeat_front_addr":"172.21.15.183:6821/13344","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.292229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.372218+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.430720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:37:59.584 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-10-15T16:37:59.584 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-10-15T16:37:59.866 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:37:59.866 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":13,"fsid":"1ed47c08-0a60-4d7f-868b-22bdfe649c24","created":"2020-10-15T16:37:47.910220+0000","modified":"2020-10-15T16:37:58.985599+0000","last_up_change":"2020-10-15T16:37:52.936722+0000","last_in_change":"2020-10-15T16:37:52.936722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:37:51.875490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-15T16:37:56.515913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bb9ac728-1df1-4164-a58e-dd20186a1168","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":13343},{"type":"v1","addr":"172.21.15.183:6801","nonce":13343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":13343},{"type":"v1","addr":"172.21.15.183:6803","nonce":13343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":13343},{"type":"v1","addr":"172.21.15.183:6807","nonce":13343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":13343},{"type":"v1","addr":"172.21.15.183:6805","nonce":13343}]},"public_addr":"172.21.15.183:6801/13343","cluster_addr":"172.21.15.183:6803/13343","heartbeat_back_addr":"172.21.15.183:6807/13343","heartbeat_front_addr":"172.21.15.183:6805/13343","state":["exists","up"]},{"osd":1,"uuid":"62b5f66a-b15d-4798-9a3b-a1c698cdf027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":13345},{"type":"v1","addr":"172.21.15.183:6809","nonce":13345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":13345},{"type":"v1","addr":"172.21.15.183:6811","nonce":13345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":13345},{"type":"v1","addr":"172.21.15.183:6815","nonce":13345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":13345},{"type":"v1","addr":"172.21.15.183:6813","nonce":13345}]},"public_addr":"172.21.15.183:6809/13345","cluster_addr":"172.21.15.183:6811/13345","heartbeat_back_addr":"172.21.15.183:6815/13345","heartbeat_front_addr":"172.21.15.183:6813/13345","state":["exists","up"]},{"osd":2,"uuid":"f3e73d60-7358-4c49-b957-d289197dac18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":13344},{"type":"v1","addr":"172.21.15.183:6817","nonce":13344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":13344},{"type":"v1","addr":"172.21.15.183:6819","nonce":13344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":13344},{"type":"v1","addr":"172.21.15.183:6823","nonce":13344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":13344},{"type":"v1","addr":"172.21.15.183:6821","nonce":13344}]},"public_addr":"172.21.15.183:6817/13344","cluster_addr":"172.21.15.183:6819/13344","heartbeat_back_addr":"172.21.15.183:6823/13344","heartbeat_front_addr":"172.21.15.183:6821/13344","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.292229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.372218+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:37:51.430720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:37:59.880 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-10-15T16:38:00.050 INFO:teuthology.orchestra.run.smithi183.stdout:34359738371 2020-10-15T16:38:00.051 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-10-15T16:38:00.221 INFO:teuthology.orchestra.run.smithi183.stdout:34359738371 2020-10-15T16:38:00.221 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-10-15T16:38:00.384 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2020-10-15T16:38:00.384 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-10-15T16:38:00.676 INFO:teuthology.orchestra.run.smithi183.stdout:34359738370 2020-10-15T16:38:00.694 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-10-15T16:38:01.695 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-10-15T16:38:01.994 INFO:teuthology.orchestra.run.smithi183.stdout:34359738370 2020-10-15T16:38:02.008 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-10-15T16:38:03.010 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-10-15T16:38:03.311 INFO:teuthology.orchestra.run.smithi183.stdout:34359738371 2020-10-15T16:38:03.325 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-10-15T16:38:03.326 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-10-15T16:38:03.613 INFO:teuthology.orchestra.run.smithi183.stdout:34359738371 2020-10-15T16:38:03.627 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2020-10-15T16:38:03.628 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-10-15T16:38:03.918 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2020-10-15T16:38:03.932 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-10-15T16:38:03.933 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-10-15T16:38:03.933 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-10-15T16:38:04.215 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:04.217 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2020-10-15T16:38:04.229 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-10-15T16:38:03.855156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.867322"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.003348+0000","last_change":"2020-10-15T16:37:58.001600+0000","last_active":"2020-10-15T16:37:58.003348+0000","last_peered":"2020-10-15T16:37:58.003348+0000","last_clean":"2020-10-15T16:37:58.003348+0000","last_became_active":"2020-10-15T16:37:58.000904+0000","last_became_peered":"2020-10-15T16:37:58.000904+0000","last_unstale":"2020-10-15T16:37:58.003348+0000","last_undegraded":"2020-10-15T16:37:58.003348+0000","last_fullsized":"2020-10-15T16:37:58.003348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004492+0000","last_change":"2020-10-15T16:37:58.002559+0000","last_active":"2020-10-15T16:37:58.004492+0000","last_peered":"2020-10-15T16:37:58.004492+0000","last_clean":"2020-10-15T16:37:58.004492+0000","last_became_active":"2020-10-15T16:37:58.002187+0000","last_became_peered":"2020-10-15T16:37:58.002187+0000","last_unstale":"2020-10-15T16:37:58.004492+0000","last_undegraded":"2020-10-15T16:37:58.004492+0000","last_fullsized":"2020-10-15T16:37:58.004492+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004956+0000","last_change":"2020-10-15T16:37:58.004316+0000","last_active":"2020-10-15T16:37:58.004956+0000","last_peered":"2020-10-15T16:37:58.004956+0000","last_clean":"2020-10-15T16:37:58.004956+0000","last_became_active":"2020-10-15T16:37:58.003993+0000","last_became_peered":"2020-10-15T16:37:58.003993+0000","last_unstale":"2020-10-15T16:37:58.004956+0000","last_undegraded":"2020-10-15T16:37:58.004956+0000","last_fullsized":"2020-10-15T16:37:58.004956+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004691+0000","last_change":"2020-10-15T16:37:58.001399+0000","last_active":"2020-10-15T16:37:58.004691+0000","last_peered":"2020-10-15T16:37:58.004691+0000","last_clean":"2020-10-15T16:37:58.004691+0000","last_became_active":"2020-10-15T16:37:58.000502+0000","last_became_peered":"2020-10-15T16:37:58.000502+0000","last_unstale":"2020-10-15T16:37:58.004691+0000","last_undegraded":"2020-10-15T16:37:58.004691+0000","last_fullsized":"2020-10-15T16:37:58.004691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:57.998287+0000","last_change":"2020-10-15T16:37:54.246248+0000","last_active":"2020-10-15T16:37:57.998287+0000","last_peered":"2020-10-15T16:37:57.998287+0000","last_clean":"2020-10-15T16:37:57.998287+0000","last_became_active":"2020-10-15T16:37:54.244907+0000","last_became_peered":"2020-10-15T16:37:54.244907+0000","last_unstale":"2020-10-15T16:37:57.998287+0000","last_undegraded":"2020-10-15T16:37:57.998287+0000","last_fullsized":"2020-10-15T16:37:57.998287+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:51.929287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:51.929287+0000","last_clean_scrub_stamp":"2020-10-15T16:37:51.929287+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004522+0000","last_change":"2020-10-15T16:37:58.002285+0000","last_active":"2020-10-15T16:37:58.004522+0000","last_peered":"2020-10-15T16:37:58.004522+0000","last_clean":"2020-10-15T16:37:58.004522+0000","last_became_active":"2020-10-15T16:37:58.001886+0000","last_became_peered":"2020-10-15T16:37:58.001886+0000","last_unstale":"2020-10-15T16:37:58.004522+0000","last_undegraded":"2020-10-15T16:37:58.004522+0000","last_fullsized":"2020-10-15T16:37:58.004522+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004415+0000","last_change":"2020-10-15T16:37:58.001727+0000","last_active":"2020-10-15T16:37:58.004415+0000","last_peered":"2020-10-15T16:37:58.004415+0000","last_clean":"2020-10-15T16:37:58.004415+0000","last_became_active":"2020-10-15T16:37:58.001165+0000","last_became_peered":"2020-10-15T16:37:58.001165+0000","last_unstale":"2020-10-15T16:37:58.004415+0000","last_undegraded":"2020-10-15T16:37:58.004415+0000","last_fullsized":"2020-10-15T16:37:58.004415+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004831+0000","last_change":"2020-10-15T16:37:58.002831+0000","last_active":"2020-10-15T16:37:58.004831+0000","last_peered":"2020-10-15T16:37:58.004831+0000","last_clean":"2020-10-15T16:37:58.004831+0000","last_became_active":"2020-10-15T16:37:58.002093+0000","last_became_peered":"2020-10-15T16:37:58.002093+0000","last_unstale":"2020-10-15T16:37:58.004831+0000","last_undegraded":"2020-10-15T16:37:58.004831+0000","last_fullsized":"2020-10-15T16:37:58.004831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.001580+0000","last_change":"2020-10-15T16:37:58.000569+0000","last_active":"2020-10-15T16:37:58.001580+0000","last_peered":"2020-10-15T16:37:58.001580+0000","last_clean":"2020-10-15T16:37:58.001580+0000","last_became_active":"2020-10-15T16:37:58.000124+0000","last_became_peered":"2020-10-15T16:37:58.000124+0000","last_unstale":"2020-10-15T16:37:58.001580+0000","last_undegraded":"2020-10-15T16:37:58.001580+0000","last_fullsized":"2020-10-15T16:37:58.001580+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]}]}]}} 2020-10-15T16:38:04.231 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-10-15T16:38:04.508 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:04.510 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2020-10-15T16:38:04.522 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-10-15T16:38:03.855156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.867322"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.003348+0000","last_change":"2020-10-15T16:37:58.001600+0000","last_active":"2020-10-15T16:37:58.003348+0000","last_peered":"2020-10-15T16:37:58.003348+0000","last_clean":"2020-10-15T16:37:58.003348+0000","last_became_active":"2020-10-15T16:37:58.000904+0000","last_became_peered":"2020-10-15T16:37:58.000904+0000","last_unstale":"2020-10-15T16:37:58.003348+0000","last_undegraded":"2020-10-15T16:37:58.003348+0000","last_fullsized":"2020-10-15T16:37:58.003348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004492+0000","last_change":"2020-10-15T16:37:58.002559+0000","last_active":"2020-10-15T16:37:58.004492+0000","last_peered":"2020-10-15T16:37:58.004492+0000","last_clean":"2020-10-15T16:37:58.004492+0000","last_became_active":"2020-10-15T16:37:58.002187+0000","last_became_peered":"2020-10-15T16:37:58.002187+0000","last_unstale":"2020-10-15T16:37:58.004492+0000","last_undegraded":"2020-10-15T16:37:58.004492+0000","last_fullsized":"2020-10-15T16:37:58.004492+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004956+0000","last_change":"2020-10-15T16:37:58.004316+0000","last_active":"2020-10-15T16:37:58.004956+0000","last_peered":"2020-10-15T16:37:58.004956+0000","last_clean":"2020-10-15T16:37:58.004956+0000","last_became_active":"2020-10-15T16:37:58.003993+0000","last_became_peered":"2020-10-15T16:37:58.003993+0000","last_unstale":"2020-10-15T16:37:58.004956+0000","last_undegraded":"2020-10-15T16:37:58.004956+0000","last_fullsized":"2020-10-15T16:37:58.004956+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004691+0000","last_change":"2020-10-15T16:37:58.001399+0000","last_active":"2020-10-15T16:37:58.004691+0000","last_peered":"2020-10-15T16:37:58.004691+0000","last_clean":"2020-10-15T16:37:58.004691+0000","last_became_active":"2020-10-15T16:37:58.000502+0000","last_became_peered":"2020-10-15T16:37:58.000502+0000","last_unstale":"2020-10-15T16:37:58.004691+0000","last_undegraded":"2020-10-15T16:37:58.004691+0000","last_fullsized":"2020-10-15T16:37:58.004691+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:57.998287+0000","last_change":"2020-10-15T16:37:54.246248+0000","last_active":"2020-10-15T16:37:57.998287+0000","last_peered":"2020-10-15T16:37:57.998287+0000","last_clean":"2020-10-15T16:37:57.998287+0000","last_became_active":"2020-10-15T16:37:54.244907+0000","last_became_peered":"2020-10-15T16:37:54.244907+0000","last_unstale":"2020-10-15T16:37:57.998287+0000","last_undegraded":"2020-10-15T16:37:57.998287+0000","last_fullsized":"2020-10-15T16:37:57.998287+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:51.929287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:51.929287+0000","last_clean_scrub_stamp":"2020-10-15T16:37:51.929287+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004522+0000","last_change":"2020-10-15T16:37:58.002285+0000","last_active":"2020-10-15T16:37:58.004522+0000","last_peered":"2020-10-15T16:37:58.004522+0000","last_clean":"2020-10-15T16:37:58.004522+0000","last_became_active":"2020-10-15T16:37:58.001886+0000","last_became_peered":"2020-10-15T16:37:58.001886+0000","last_unstale":"2020-10-15T16:37:58.004522+0000","last_undegraded":"2020-10-15T16:37:58.004522+0000","last_fullsized":"2020-10-15T16:37:58.004522+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004415+0000","last_change":"2020-10-15T16:37:58.001727+0000","last_active":"2020-10-15T16:37:58.004415+0000","last_peered":"2020-10-15T16:37:58.004415+0000","last_clean":"2020-10-15T16:37:58.004415+0000","last_became_active":"2020-10-15T16:37:58.001165+0000","last_became_peered":"2020-10-15T16:37:58.001165+0000","last_unstale":"2020-10-15T16:37:58.004415+0000","last_undegraded":"2020-10-15T16:37:58.004415+0000","last_fullsized":"2020-10-15T16:37:58.004415+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.004831+0000","last_change":"2020-10-15T16:37:58.002831+0000","last_active":"2020-10-15T16:37:58.004831+0000","last_peered":"2020-10-15T16:37:58.004831+0000","last_clean":"2020-10-15T16:37:58.004831+0000","last_became_active":"2020-10-15T16:37:58.002093+0000","last_became_peered":"2020-10-15T16:37:58.002093+0000","last_unstale":"2020-10-15T16:37:58.004831+0000","last_undegraded":"2020-10-15T16:37:58.004831+0000","last_fullsized":"2020-10-15T16:37:58.004831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:37:58.001580+0000","last_change":"2020-10-15T16:37:58.000569+0000","last_active":"2020-10-15T16:37:58.001580+0000","last_peered":"2020-10-15T16:37:58.001580+0000","last_clean":"2020-10-15T16:37:58.001580+0000","last_became_active":"2020-10-15T16:37:58.000124+0000","last_became_peered":"2020-10-15T16:37:58.000124+0000","last_unstale":"2020-10-15T16:37:58.001580+0000","last_undegraded":"2020-10-15T16:37:58.001580+0000","last_fullsized":"2020-10-15T16:37:58.001580+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:37:56.970676+0000","last_clean_scrub_stamp":"2020-10-15T16:37:56.970676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]}]}]}} 2020-10-15T16:38:04.523 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-10-15T16:38:04.524 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-10-15T16:38:04.524 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-10-15T16:38:04.524 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-10-15T16:38:04.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:04.877 INFO:teuthology.orchestra.run.smithi183.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-10-15T16:38:04.877 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-10-15T16:38:04.878 INFO:teuthology.run_tasks:Running task ceph... 2020-10-15T16:38:04.891 INFO:tasks.ceph:Creating ceph cluster c2... 2020-10-15T16:38:04.891 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-10-15T16:38:04.892 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi/5527098', 'branch': 'wip-yuval-cls-fifo', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '5527098', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-10-15_12:17:42-rgw-wip-yuval-cls-fifo-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuval-cls-fifo'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuval-cls-fifo', 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuval-cls-fifo', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '417a7e57ff62f4d3de1c307caeb5ea5d04efadf1', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.15069', 'targets': {'smithi183.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyyo5jDAgNCY2MAVJg7aLSIIrwNjaSiAqJyM9dxk4rMvo/VKE1meyht5fMhq+77fogrBP3Y8nfI9WRD1WvifWzXuLSftj7hTCcPEHJDRTUwBNeyg2FHMne2CrEAERvBNvkJkIRp79EeozjMGpNZ4Dxz8cG0kGiaUcCaWSprB3pLuCtg0oHKJ9M/C98g4fI4uXWmyfQNx5XgJJSJ5EOJcNw5jWS1xoq61U8APotg6FyxMSTh/YWpDb7wf1GHgxNFvQS+MhgWvZxR25aJ4wYev6HPy6neeVTlnbJH1SqCn7ChVswNcyVDMR6a5OTd4QMDLs+TIk7cnNN0TLF0mvlUCr3', 'smithi038.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1tZHJ9OhSMJ524FtRmYYH9mKdurGuoC1xzwIc+OnxZ3GQCn/VLl87OsLJ5F7GkCFOSabal/LT3fLJ+GztdJk8iEFZjn/LVrGWshqgyATInaHRf4KJdVRNBS2Yhvg9916OMVHQYHUbUPFwKjfsqCVQmEBc0dAYvsJH8IqpR7t0ByfgsY0A5MyGUdo/qHiq7j3yYU9yIr4m+mPiK6x2d3RGgkR6OO9zU86kRU6R8zBN7Yne6UNBnR3WWsEivDppY1cAb9Ty1XhLMRpiEd/cGxVRidv+RY8cXCFzf9hg6YPnkcSnszdXVEYXod/2Oau63yeM5L70mTdah/Gmr4E4qRP1'}} 2020-10-15T16:38:04.893 INFO:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-10-15T16:38:04.895 INFO:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-10-15T16:38:04.902 INFO:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2020-10-15T16:38:04.942 INFO:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2020-10-15T16:38:04.958 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:04.958 INFO:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2020-10-15T16:38:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-10-15T16:38:05.002 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_1 2020-10-15T16:38:05.002 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_2 2020-10-15T16:38:05.002 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_3 2020-10-15T16:38:05.002 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/vg_nvme/lv_4 2020-10-15T16:38:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:72 bytes copied, 0.000180617 s, 399 kB/s 2020-10-15T16:38:05.003 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-10-15T16:38:05.003 INFO:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-10-15T16:38:05.051 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-10-15T16:38:05.051 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:38:05.051 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 342 Links: 1 2020-10-15T16:38:05.051 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:38:05.052 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2020-10-15 16:35:10.329219663 +0000 2020-10-15T16:38:05.052 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2020-10-15 16:35:10.217163683 +0000 2020-10-15T16:38:05.052 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2020-10-15 16:35:10.217163683 +0000 2020-10-15T16:38:05.052 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2020-10-15T16:38:05.061 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2020-10-15T16:38:05.061 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2020-10-15T16:38:05.061 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000451585 s, 1.1 MB/s 2020-10-15T16:38:05.065 INFO:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-10-15T16:38:05.115 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-10-15T16:38:05.115 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:38:05.115 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 349 Links: 1 2020-10-15T16:38:05.115 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:38:05.116 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2020-10-15 16:35:10.577343619 +0000 2020-10-15T16:38:05.116 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2020-10-15 16:35:10.465287639 +0000 2020-10-15T16:38:05.116 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2020-10-15 16:35:10.465287639 +0000 2020-10-15T16:38:05.116 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2020-10-15T16:38:05.125 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2020-10-15T16:38:05.125 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2020-10-15T16:38:05.125 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000489328 s, 1.0 MB/s 2020-10-15T16:38:05.129 INFO:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-10-15T16:38:05.179 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-10-15T16:38:05.179 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:38:05.179 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 355 Links: 1 2020-10-15T16:38:05.180 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:38:05.180 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2020-10-15 16:35:10.841475572 +0000 2020-10-15T16:38:05.180 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2020-10-15 16:35:10.741425590 +0000 2020-10-15T16:38:05.180 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2020-10-15 16:35:10.741425590 +0000 2020-10-15T16:38:05.180 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2020-10-15T16:38:05.189 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2020-10-15T16:38:05.189 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2020-10-15T16:38:05.189 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000385525 s, 1.3 MB/s 2020-10-15T16:38:05.193 INFO:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-10-15T16:38:05.243 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-10-15T16:38:05.243 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-15T16:38:05.243 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 361 Links: 1 2020-10-15T16:38:05.243 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-15T16:38:05.244 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2020-10-15 16:35:10.985547546 +0000 2020-10-15T16:38:05.244 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2020-10-15 16:35:10.981545547 +0000 2020-10-15T16:38:05.244 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2020-10-15 16:35:10.981545547 +0000 2020-10-15T16:38:05.244 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2020-10-15T16:38:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2020-10-15T16:38:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2020-10-15T16:38:05.253 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000422881 s, 1.2 MB/s 2020-10-15T16:38:05.257 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:38:05.257 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-15T16:38:05.257 INFO:tasks.ceph:Generating config... 2020-10-15T16:38:05.259 INFO:tasks.ceph:[client] debug rgw = 20 2020-10-15T16:38:05.259 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-10-15T16:38:05.259 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-10-15T16:38:05.259 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-15T16:38:05.260 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-10-15T16:38:05.260 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-10-15T16:38:05.260 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-10-15T16:38:05.260 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-10-15T16:38:05.260 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-10-15T16:38:05.261 INFO:tasks.ceph:[client] setgroup = ceph 2020-10-15T16:38:05.261 INFO:tasks.ceph:[client] setuser = ceph 2020-10-15T16:38:05.261 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-10-15T16:38:05.261 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-10-15T16:38:05.261 INFO:tasks.ceph:[mgr] debug ms = 1 2020-10-15T16:38:05.262 INFO:tasks.ceph:[mon] debug mon = 20 2020-10-15T16:38:05.262 INFO:tasks.ceph:[mon] debug ms = 1 2020-10-15T16:38:05.262 INFO:tasks.ceph:[mon] debug paxos = 20 2020-10-15T16:38:05.262 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-10-15T16:38:05.262 INFO:tasks.ceph:[osd] debug filestore = 20 2020-10-15T16:38:05.263 INFO:tasks.ceph:[osd] debug journal = 20 2020-10-15T16:38:05.263 INFO:tasks.ceph:[osd] debug ms = 20 2020-10-15T16:38:05.263 INFO:tasks.ceph:[osd] debug osd = 25 2020-10-15T16:38:05.263 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-10-15T16:38:05.263 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-10-15T16:38:05.264 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-10-15T16:38:05.264 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-10-15T16:38:05.264 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-10-15T16:38:05.264 INFO:tasks.ceph:Setting up c2.mon.a... 2020-10-15T16:38:05.265 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-10-15T16:38:05.333 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/c2.keyring 2020-10-15T16:38:05.335 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-10-15T16:38:05.413 INFO:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-15T16:38:05.471 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.38')] 2020-10-15T16:38:05.471 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.38', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-10-15T16:38:05.472 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:05.473 INFO:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-15T16:38:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-15T16:38:05.518 INFO:teuthology.orchestra.run.smithi038.stderr:6+1 records in 2020-10-15T16:38:05.518 INFO:teuthology.orchestra.run.smithi038.stderr:6+1 records out 2020-10-15T16:38:05.518 INFO:teuthology.orchestra.run.smithi038.stderr:3100 bytes (3.1 kB, 3.0 KiB) copied, 0.00011202 s, 27.7 MB/s 2020-10-15T16:38:05.519 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --print /home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:38:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:38:05.591 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:05.591 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid 95640a16-2ac6-41cc-ab44-74701d09dc50 2020-10-15T16:38:05.592 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2020-10-15T16:38:05.592 INFO:teuthology.orchestra.run.smithi038.stdout:fsid 95640a16-2ac6-41cc-ab44-74701d09dc50 2020-10-15T16:38:05.592 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2020-10-15T16:38:05.594798+0000 2020-10-15T16:38:05.592 INFO:teuthology.orchestra.run.smithi038.stdout:created 2020-10-15T16:38:05.594798+0000 2020-10-15T16:38:05.592 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 0 (unknown) 2020-10-15T16:38:05.592 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 1 2020-10-15T16:38:05.593 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2020-10-15T16:38:05.593 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-10-15T16:38:05.594 INFO:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-10-15T16:38:05.641 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 95640a16-2ac6-41cc-ab44-74701d09dc50... 2020-10-15T16:38:05.643 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-10-15T16:38:05.686 INFO:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-10-15T16:38:05.719 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2020-10-15T16:38:05.719 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2020-10-15T16:38:05.719 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-15T16:38:05.719 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2020-10-15T16:38:05.719 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.720 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2020-10-15T16:38:05.720 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.720 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2020-10-15T16:38:05.720 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.720 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2020-10-15T16:38:05.720 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.721 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2020-10-15T16:38:05.722 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-15T16:38:05.723 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2020-10-15T16:38:05.724 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = 172.21.15.38 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: osd heartbeat grace = 40 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 95640a16-2ac6-41cc-ab44-74701d09dc50 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2020-10-15T16:38:05.725 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2020-10-15T16:38:05.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.727 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2020-10-15T16:38:05.728 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: debug filestore = 20 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: debug journal = 20 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 20 2020-10-15T16:38:05.729 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 25 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: osd fast shutdown = False 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: rocksdb delete range threshold = 0 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2020-10-15T16:38:05.730 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2020-10-15T16:38:05.731 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2020-10-15T16:38:05.732 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd crush smoke test = False 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2020-10-15T16:38:05.733 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: debug rgw = 20 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt require ssl = False 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt s3 kms backend = testing 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: rgw curl low speed time = 300 2020-10-15T16:38:05.734 INFO:teuthology.orchestra.run.smithi038.stdout: rgw data log num shards = 4 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout: rgw md log max shards = 4 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout: rgw sync log trim interval = 0 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout: rgw sync obj etag verify = True 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout: setgroup = ceph 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout: setuser = ceph 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2020-10-15T16:38:05.735 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_omap_entries_per_request = 10 2020-10-15T16:38:05.736 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2020-10-15T16:38:05.736 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_omap_entries_per_request = 1000 2020-10-15T16:38:05.736 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2020-10-15T16:38:05.736 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_omap_entries_per_request = 10000 2020-10-15T16:38:05.737 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2020-10-15T16:38:05.737 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2020-10-15T16:38:05.738 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.739 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2020-10-15T16:38:05.740 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2020-10-15T16:38:05.741 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2020-10-15T16:38:05.742 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = 172.21.15.38 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: osd heartbeat grace = 40 2020-10-15T16:38:05.743 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 95640a16-2ac6-41cc-ab44-74701d09dc50 2020-10-15T16:38:05.744 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.744 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2020-10-15T16:38:05.744 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2020-10-15T16:38:05.744 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.744 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2020-10-15T16:38:05.744 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2020-10-15T16:38:05.745 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2020-10-15T16:38:05.746 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.746 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2020-10-15T16:38:05.746 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2020-10-15T16:38:05.746 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.746 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2020-10-15T16:38:05.746 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.747 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2020-10-15T16:38:05.747 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.747 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2020-10-15T16:38:05.747 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.747 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2020-10-15T16:38:05.747 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: debug filestore = 20 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: debug journal = 20 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 20 2020-10-15T16:38:05.748 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 25 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout: osd fast shutdown = False 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout: rocksdb delete range threshold = 0 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2020-10-15T16:38:05.749 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2020-10-15T16:38:05.750 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2020-10-15T16:38:05.751 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd crush smoke test = False 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2020-10-15T16:38:05.752 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: debug rgw = 20 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: rgw crypt require ssl = False 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: rgw crypt s3 kms backend = testing 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-15T16:38:05.753 INFO:teuthology.orchestra.run.smithi183.stdout: rgw curl low speed time = 300 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: rgw data log num shards = 4 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: rgw md log max shards = 4 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: rgw sync log trim interval = 0 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: rgw sync obj etag verify = True 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: setgroup = ceph 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: setuser = ceph 2020-10-15T16:38:05.754 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2020-10-15T16:38:05.755 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2020-10-15T16:38:05.755 INFO:teuthology.orchestra.run.smithi183.stdout: osd_max_omap_entries_per_request = 10 2020-10-15T16:38:05.755 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2020-10-15T16:38:05.755 INFO:teuthology.orchestra.run.smithi183.stdout: osd_max_omap_entries_per_request = 1000 2020-10-15T16:38:05.755 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2020-10-15T16:38:05.755 INFO:teuthology.orchestra.run.smithi183.stdout: osd_max_omap_entries_per_request = 10000 2020-10-15T16:38:05.756 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-10-15T16:38:05.756 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-10-15T16:38:05.826 INFO:tasks.ceph:Copying monmap to all nodes... 2020-10-15T16:38:05.827 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:05.827 INFO:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-10-15T16:38:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-10-15T16:38:05.833 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.] 2020-10-15T16:38:05.833 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDteohfJ4SzGBAAE3vlSpGF/qIJ+h2r5kNvuA== 2020-10-15T16:38:05.833 INFO:teuthology.orchestra.run.smithi038.stdout:[client.admin] 2020-10-15T16:38:05.834 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDteohfRnBUMRAAwqy1Akchxo+CfzUR91AyXQ== 2020-10-15T16:38:05.834 INFO:teuthology.orchestra.run.smithi038.stdout: caps mds = "allow *" 2020-10-15T16:38:05.834 INFO:teuthology.orchestra.run.smithi038.stdout: caps mgr = "allow *" 2020-10-15T16:38:05.834 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "allow *" 2020-10-15T16:38:05.834 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "allow *" 2020-10-15T16:38:05.834 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:05.835 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:05.835 INFO:teuthology.orchestra.run.smithi038.stderr:206 bytes copied, 0.000162537 s, 1.3 MB/s 2020-10-15T16:38:05.835 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:05.835 INFO:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-10-15T16:38:05.880 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-10-15T16:38:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: ��d 2020-10-15T16:38:05.882 INFO:teuthology.orchestra.run.smithi038.stdout:*�A̫Dtp �P�z�_I�s#�z�_I�s#�aXa �&��&a 2020-10-15T16:38:05.882 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:05.882 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:05.882 INFO:teuthology.orchestra.run.smithi038.stderr:201 bytes copied, 0.000163085 s, 1.2 MB/s 2020-10-15T16:38:05.883 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2020-10-15T16:38:05.883 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:38:05.883 INFO:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/c2.keyring 2020-10-15T16:38:05.883 INFO:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-15T16:38:05.888 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-10-15T16:38:05.898 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:38:05.898 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:38:05.898 INFO:teuthology.orchestra.run.smithi183.stderr:206 bytes copied, 7.1914e-05 s, 2.9 MB/s 2020-10-15T16:38:05.899 INFO:teuthology.orchestra.run.smithi183.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-15T16:38:05.908 INFO:teuthology.orchestra.run.smithi183:> set -ex 2020-10-15T16:38:05.909 INFO:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:05.956 INFO:teuthology.orchestra.run.smithi183.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:05.957 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2020-10-15T16:38:05.957 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2020-10-15T16:38:05.957 INFO:teuthology.orchestra.run.smithi183.stderr:201 bytes copied, 7.551e-05 s, 2.7 MB/s 2020-10-15T16:38:05.958 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:38:05.958 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:05.958 INFO:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/c2.keyring 2020-10-15T16:38:05.959 INFO:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-15T16:38:05.963 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-10-15T16:38:05.973 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:05.973 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:05.973 INFO:teuthology.orchestra.run.smithi038.stderr:206 bytes copied, 0.00013136 s, 1.6 MB/s 2020-10-15T16:38:05.974 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-15T16:38:05.984 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:05.984 INFO:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:06.029 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:06.030 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:06.031 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:06.031 INFO:teuthology.orchestra.run.smithi038.stderr:201 bytes copied, 0.000129107 s, 1.6 MB/s 2020-10-15T16:38:06.031 INFO:tasks.ceph:Setting up mon nodes... 2020-10-15T16:38:06.032 INFO:tasks.ceph:Setting up mgr nodes... 2020-10-15T16:38:06.033 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-10-15T16:38:06.113 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-10-15T16:38:06.116 INFO:tasks.ceph:Setting up mds nodes... 2020-10-15T16:38:06.116 INFO:tasks.ceph_client:Setting up client nodes... 2020-10-15T16:38:06.117 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-10-15T16:38:06.191 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/c2.client.0.keyring 2020-10-15T16:38:06.203 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-10-15T16:38:06.277 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/c2.client.1.keyring 2020-10-15T16:38:06.289 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-10-15T16:38:06.290 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-15T16:38:06.290 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-10-15T16:38:06.347 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:38:06.348 INFO:tasks.ceph:role: c2.osd.0 2020-10-15T16:38:06.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:38:06.348 INFO:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-10-15T16:38:07.721 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-10-15T16:38:07.721 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-15T16:38:07.721 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-15T16:38:07.722 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-15T16:38:07.722 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2020-10-15T16:38:07.722 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-15T16:38:07.722 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-15T16:38:07.722 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-15T16:38:07.722 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-15T16:38:07.723 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2020-10-15T16:38:07.724 INFO:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-10-15T16:38:07.745 INFO:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-10-15T16:38:07.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2020-10-15T16:38:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-15T16:38:07.801 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-10-15T16:38:07.858 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:38:07.858 INFO:tasks.ceph:role: c2.osd.1 2020-10-15T16:38:07.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:38:07.859 INFO:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-10-15T16:38:09.239 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-10-15T16:38:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-15T16:38:09.240 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-15T16:38:09.241 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2020-10-15T16:38:09.242 INFO:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-10-15T16:38:09.264 INFO:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-10-15T16:38:09.319 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2020-10-15T16:38:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-15T16:38:09.320 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-10-15T16:38:09.374 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-15T16:38:09.375 INFO:tasks.ceph:role: c2.osd.2 2020-10-15T16:38:09.375 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2020-10-15T16:38:09.376 INFO:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-10-15T16:38:10.754 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-10-15T16:38:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-15T16:38:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-15T16:38:10.755 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-15T16:38:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2020-10-15T16:38:10.755 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-15T16:38:10.755 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-15T16:38:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-15T16:38:10.755 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-15T16:38:10.756 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2020-10-15T16:38:10.757 INFO:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-10-15T16:38:10.784 INFO:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-10-15T16:38:10.836 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2020-10-15T16:38:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-15T16:38:10.837 INFO:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:10.921+0000 7f571d5e7d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-10-15T16:38:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:10.921+0000 7f571d5e7d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-10-15T16:38:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:10.921+0000 7f571d5e7d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-10-15T16:38:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:10.921+0000 7f571d5e7d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-10-15T16:38:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:10.921+0000 7f571d5e7d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-10-15T16:38:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:10.925+0000 7f571d5e7d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-10-15T16:38:13.117 INFO:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-10-15T16:38:13.175 INFO:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:13.254 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:13.253+0000 7f56340ded80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-10-15T16:38:13.255 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:13.253+0000 7f56340ded80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-10-15T16:38:13.255 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:13.253+0000 7f56340ded80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-10-15T16:38:13.255 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:13.253+0000 7f56340ded80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-10-15T16:38:13.255 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:13.253+0000 7f56340ded80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-10-15T16:38:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:13.253+0000 7f56340ded80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-10-15T16:38:15.448 INFO:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-10-15T16:38:15.508 INFO:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:15.589+0000 7f5928e64d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-10-15T16:38:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:15.589+0000 7f5928e64d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-10-15T16:38:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:15.589+0000 7f5928e64d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-10-15T16:38:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:15.589+0000 7f5928e64d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-10-15T16:38:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:15.589+0000 7f5928e64d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-10-15T16:38:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:15.589+0000 7f5928e64d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-10-15T16:38:17.770 INFO:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-10-15T16:38:17.827 INFO:tasks.ceph:Reading keys from all nodes... 2020-10-15T16:38:17.828 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:17.828 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-10-15T16:38:17.872 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-10-15T16:38:17.881 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2020-10-15T16:38:17.881 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDueohf1YX7BhAAOYC0W28m5gtGr/u5EkR6Hg== 2020-10-15T16:38:17.882 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:17.882 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:17.882 INFO:teuthology.orchestra.run.smithi038.stderr:56 bytes copied, 0.000181007 s, 309 kB/s 2020-10-15T16:38:17.883 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:17.883 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-10-15T16:38:17.928 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-10-15T16:38:17.937 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2020-10-15T16:38:17.937 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDyeohfZRhBNxAA9/H1o0r04air7VKIjlIhuA== 2020-10-15T16:38:17.937 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:17.937 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:17.938 INFO:teuthology.orchestra.run.smithi038.stderr:56 bytes copied, 0.000166734 s, 336 kB/s 2020-10-15T16:38:17.938 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:17.939 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-10-15T16:38:17.984 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-10-15T16:38:17.993 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2020-10-15T16:38:17.994 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQD1eohfjQ9iDxAAInOP0mQflde4omLUoj6Whw== 2020-10-15T16:38:17.994 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:17.994 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:17.994 INFO:teuthology.orchestra.run.smithi038.stderr:56 bytes copied, 0.000155734 s, 360 kB/s 2020-10-15T16:38:17.995 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:17.995 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-10-15T16:38:18.040 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-10-15T16:38:18.049 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2020-10-15T16:38:18.050 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQD3eohfQ1BuIxAArU0m3Wqg3EbLFOpCuDmM8Q== 2020-10-15T16:38:18.050 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:18.050 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:18.050 INFO:teuthology.orchestra.run.smithi038.stderr:56 bytes copied, 0.000165558 s, 338 kB/s 2020-10-15T16:38:18.051 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:18.052 INFO:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-10-15T16:38:18.096 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-10-15T16:38:18.098 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2020-10-15T16:38:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDueohfG2yiCxAA6zWHcrlCOLeu3iSW2+GKng== 2020-10-15T16:38:18.098 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:18.099 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:18.099 INFO:teuthology.orchestra.run.smithi038.stderr:59 bytes copied, 0.000166377 s, 355 kB/s 2020-10-15T16:38:18.099 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:18.099 INFO:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-10-15T16:38:18.144 INFO:teuthology.orchestra.run.smithi038.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-10-15T16:38:18.146 INFO:teuthology.orchestra.run.smithi038.stdout:[client.1] 2020-10-15T16:38:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDueohfY4fBEBAAaOQHpsd7mreOHeq18b8d1w== 2020-10-15T16:38:18.146 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:18.146 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:18.147 INFO:teuthology.orchestra.run.smithi038.stderr:59 bytes copied, 0.000162893 s, 362 kB/s 2020-10-15T16:38:18.147 INFO:tasks.ceph:Adding keys to all mons... 2020-10-15T16:38:18.147 INFO:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/c2.keyring 2020-10-15T16:38:18.200 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2020-10-15T16:38:18.200 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDueohf1YX7BhAAOYC0W28m5gtGr/u5EkR6Hg== 2020-10-15T16:38:18.200 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2020-10-15T16:38:18.200 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDyeohfZRhBNxAA9/H1o0r04air7VKIjlIhuA== 2020-10-15T16:38:18.200 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2020-10-15T16:38:18.201 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQD1eohfjQ9iDxAAInOP0mQflde4omLUoj6Whw== 2020-10-15T16:38:18.201 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2020-10-15T16:38:18.201 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQD3eohfQ1BuIxAArU0m3Wqg3EbLFOpCuDmM8Q== 2020-10-15T16:38:18.201 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2020-10-15T16:38:18.202 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDueohfG2yiCxAA6zWHcrlCOLeu3iSW2+GKng== 2020-10-15T16:38:18.202 INFO:teuthology.orchestra.run.smithi038.stdout:[client.1] 2020-10-15T16:38:18.202 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDueohfY4fBEBAAaOQHpsd7mreOHeq18b8d1w== 2020-10-15T16:38:18.203 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-10-15T16:38:18.286 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-15T16:38:18.370 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-15T16:38:18.457 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-15T16:38:18.542 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-15T16:38:18.629 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-15T16:38:18.717 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-10-15T16:38:18.718 INFO:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-10-15T16:38:18.727 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-10-15T16:38:18.827 INFO:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-10-15T16:38:18.840 INFO:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-10-15T16:38:18.889 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-10-15T16:38:18.890 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-10-15T16:38:18.891 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-10-15T16:38:18.934 INFO:tasks.ceph.c2.mon.a:Started 2020-10-15T16:38:18.934 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-10-15T16:38:18.934 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-10-15T16:38:18.935 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-10-15T16:38:18.937 INFO:tasks.ceph.c2.mgr.x:Started 2020-10-15T16:38:18.937 INFO:tasks.ceph:Setting crush tunables to default 2020-10-15T16:38:18.937 INFO:teuthology.orchestra.run.smithi038:> sudo ceph --cluster c2 osd crush tunables default 2020-10-15T16:38:19.423 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2020-10-15T16:38:19.480 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-10-15T16:38:19.480 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:19.480 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-10-15T16:38:19.485 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-10-15T16:38:19.493 INFO:teuthology.orchestra.run.smithi038.stdout:5b5187a2-4591-4988-b6dc-17bdd5679d71 2020-10-15T16:38:19.494 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:19.494 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:19.494 INFO:teuthology.orchestra.run.smithi038.stderr:37 bytes copied, 0.000132383 s, 279 kB/s 2020-10-15T16:38:19.495 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:19.495 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-10-15T16:38:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-10-15T16:38:19.548 INFO:teuthology.orchestra.run.smithi038.stdout:809c9565-3e26-4295-b0d3-2b0414d350f7 2020-10-15T16:38:19.548 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:19.548 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:19.548 INFO:teuthology.orchestra.run.smithi038.stderr:37 bytes copied, 0.00017624 s, 210 kB/s 2020-10-15T16:38:19.549 INFO:teuthology.orchestra.run.smithi038:> set -ex 2020-10-15T16:38:19.549 INFO:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-10-15T16:38:19.596 INFO:teuthology.orchestra.run.smithi038.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-10-15T16:38:19.604 INFO:teuthology.orchestra.run.smithi038.stdout:9433d174-9f6b-44d6-817a-8c8465a2a0d7 2020-10-15T16:38:19.604 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2020-10-15T16:38:19.604 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2020-10-15T16:38:19.605 INFO:teuthology.orchestra.run.smithi038.stderr:37 bytes copied, 0.000206277 s, 179 kB/s 2020-10-15T16:38:19.606 INFO:teuthology.orchestra.run.smithi038:> sudo ceph --cluster c2 osd new 5b5187a2-4591-4988-b6dc-17bdd5679d71 0 2020-10-15T16:38:19.870 INFO:teuthology.orchestra.run.smithi038.stdout:0 2020-10-15T16:38:19.883 INFO:teuthology.orchestra.run.smithi038:> sudo ceph --cluster c2 osd new 809c9565-3e26-4295-b0d3-2b0414d350f7 1 2020-10-15T16:38:20.091 INFO:teuthology.orchestra.run.smithi038.stdout:1 2020-10-15T16:38:20.103 INFO:teuthology.orchestra.run.smithi038:> sudo ceph --cluster c2 osd new 9433d174-9f6b-44d6-817a-8c8465a2a0d7 2 2020-10-15T16:38:20.317 INFO:teuthology.orchestra.run.smithi038.stdout:2 2020-10-15T16:38:20.334 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-10-15T16:38:20.335 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-10-15T16:38:20.336 INFO:tasks.ceph.c2.osd.0:Started 2020-10-15T16:38:20.337 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-10-15T16:38:20.337 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-10-15T16:38:20.338 INFO:tasks.ceph.c2.osd.1:Started 2020-10-15T16:38:20.339 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-10-15T16:38:20.339 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-10-15T16:38:20.341 INFO:tasks.ceph.c2.osd.2:Started 2020-10-15T16:38:20.341 INFO:tasks.ceph:Waiting for OSDs to come up 2020-10-15T16:38:20.648 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-10-15T16:38:20.841 INFO:teuthology.misc.health.smithi038.stdout: 2020-10-15T16:38:20.842 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":5,"fsid":"95640a16-2ac6-41cc-ab44-74701d09dc50","created":"2020-10-15T16:38:19.059495+0000","modified":"2020-10-15T16:38:20.316472+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"5b5187a2-4591-4988-b6dc-17bdd5679d71","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"809c9565-3e26-4295-b0d3-2b0414d350f7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9433d174-9f6b-44d6-817a-8c8465a2a0d7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:38:20.852 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-10-15T16:38:20.942 INFO:tasks.ceph.c2.osd.1.smithi038.stderr:2020-10-15T16:38:20.941+0000 7fc5a6f42d80 -1 Falling back to public interface 2020-10-15T16:38:20.943 INFO:tasks.ceph.c2.osd.2.smithi038.stderr:2020-10-15T16:38:20.941+0000 7fe69d2b3d80 -1 Falling back to public interface 2020-10-15T16:38:20.943 INFO:tasks.ceph.c2.osd.0.smithi038.stderr:2020-10-15T16:38:20.941+0000 7fe6c9e1dd80 -1 Falling back to public interface 2020-10-15T16:38:21.334 INFO:tasks.ceph.c2.osd.2.smithi038.stderr:2020-10-15T16:38:21.333+0000 7fe69d2b3d80 -1 osd.2 0 log_to_monitors {default=true} 2020-10-15T16:38:21.359 INFO:tasks.ceph.c2.osd.0.smithi038.stderr:2020-10-15T16:38:21.357+0000 7fe6c9e1dd80 -1 osd.0 0 log_to_monitors {default=true} 2020-10-15T16:38:21.579 INFO:tasks.ceph.c2.osd.1.smithi038.stderr:2020-10-15T16:38:21.577+0000 7fc5a6f42d80 -1 osd.1 0 log_to_monitors {default=true} 2020-10-15T16:38:23.083 INFO:tasks.ceph.c2.osd.2.smithi038.stderr:2020-10-15T16:38:23.081+0000 7fe69477e700 -1 osd.2 0 waiting for initial osdmap 2020-10-15T16:38:23.083 INFO:tasks.ceph.c2.osd.0.smithi038.stderr:2020-10-15T16:38:23.081+0000 7fe6c12e8700 -1 osd.0 0 waiting for initial osdmap 2020-10-15T16:38:23.083 INFO:tasks.ceph.c2.osd.1.smithi038.stderr:2020-10-15T16:38:23.081+0000 7fc59e40d700 -1 osd.1 0 waiting for initial osdmap 2020-10-15T16:38:27.157 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-10-15T16:38:27.443 INFO:teuthology.misc.health.smithi038.stdout: 2020-10-15T16:38:27.443 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"95640a16-2ac6-41cc-ab44-74701d09dc50","created":"2020-10-15T16:38:19.059495+0000","modified":"2020-10-15T16:38:26.123551+0000","last_up_change":"2020-10-15T16:38:24.098780+0000","last_in_change":"2020-10-15T16:38:24.098780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:38:23.114481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"5b5187a2-4591-4988-b6dc-17bdd5679d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":13355},{"type":"v1","addr":"172.21.15.38:6806","nonce":13355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":13355},{"type":"v1","addr":"172.21.15.38:6812","nonce":13355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6821","nonce":13355},{"type":"v1","addr":"172.21.15.38:6823","nonce":13355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":13355},{"type":"v1","addr":"172.21.15.38:6818","nonce":13355}]},"public_addr":"172.21.15.38:6806/13355","cluster_addr":"172.21.15.38:6812/13355","heartbeat_back_addr":"172.21.15.38:6823/13355","heartbeat_front_addr":"172.21.15.38:6818/13355","state":["exists","up"]},{"osd":1,"uuid":"809c9565-3e26-4295-b0d3-2b0414d350f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":13356},{"type":"v1","addr":"172.21.15.38:6801","nonce":13356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":13356},{"type":"v1","addr":"172.21.15.38:6807","nonce":13356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":13356},{"type":"v1","addr":"172.21.15.38:6819","nonce":13356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":13356},{"type":"v1","addr":"172.21.15.38:6813","nonce":13356}]},"public_addr":"172.21.15.38:6801/13356","cluster_addr":"172.21.15.38:6807/13356","heartbeat_back_addr":"172.21.15.38:6819/13356","heartbeat_front_addr":"172.21.15.38:6813/13356","state":["exists","up"]},{"osd":2,"uuid":"9433d174-9f6b-44d6-817a-8c8465a2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":13357},{"type":"v1","addr":"172.21.15.38:6804","nonce":13357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":13357},{"type":"v1","addr":"172.21.15.38:6811","nonce":13357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":13357},{"type":"v1","addr":"172.21.15.38:6822","nonce":13357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":13357},{"type":"v1","addr":"172.21.15.38:6817","nonce":13357}]},"public_addr":"172.21.15.38:6804/13357","cluster_addr":"172.21.15.38:6811/13357","heartbeat_back_addr":"172.21.15.38:6822/13357","heartbeat_front_addr":"172.21.15.38:6817/13357","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.332536+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.619276+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.340269+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:38:27.456 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-10-15T16:38:27.456 INFO:tasks.ceph:Creating RBD pool 2020-10-15T16:38:27.457 INFO:teuthology.orchestra.run.smithi038:> sudo ceph --cluster c2 osd pool create rbd 8 2020-10-15T16:38:28.151 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2020-10-15T16:38:28.165 INFO:teuthology.orchestra.run.smithi038:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-10-15T16:38:29.166 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rbd' on pool 'rbd' 2020-10-15T16:38:29.179 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-10-15T16:38:29.179 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-10-15T16:38:29.180 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-10-15T16:38:29.468 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:29.469 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":12,"fsid":"95640a16-2ac6-41cc-ab44-74701d09dc50","created":"2020-10-15T16:38:19.059495+0000","modified":"2020-10-15T16:38:29.148564+0000","last_up_change":"2020-10-15T16:38:24.098780+0000","last_in_change":"2020-10-15T16:38:24.098780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:38:23.114481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-15T16:38:27.750568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5b5187a2-4591-4988-b6dc-17bdd5679d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":13355},{"type":"v1","addr":"172.21.15.38:6806","nonce":13355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":13355},{"type":"v1","addr":"172.21.15.38:6812","nonce":13355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6821","nonce":13355},{"type":"v1","addr":"172.21.15.38:6823","nonce":13355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":13355},{"type":"v1","addr":"172.21.15.38:6818","nonce":13355}]},"public_addr":"172.21.15.38:6806/13355","cluster_addr":"172.21.15.38:6812/13355","heartbeat_back_addr":"172.21.15.38:6823/13355","heartbeat_front_addr":"172.21.15.38:6818/13355","state":["exists","up"]},{"osd":1,"uuid":"809c9565-3e26-4295-b0d3-2b0414d350f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":13356},{"type":"v1","addr":"172.21.15.38:6801","nonce":13356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":13356},{"type":"v1","addr":"172.21.15.38:6807","nonce":13356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":13356},{"type":"v1","addr":"172.21.15.38:6819","nonce":13356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":13356},{"type":"v1","addr":"172.21.15.38:6813","nonce":13356}]},"public_addr":"172.21.15.38:6801/13356","cluster_addr":"172.21.15.38:6807/13356","heartbeat_back_addr":"172.21.15.38:6819/13356","heartbeat_front_addr":"172.21.15.38:6813/13356","state":["exists","up"]},{"osd":2,"uuid":"9433d174-9f6b-44d6-817a-8c8465a2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":13357},{"type":"v1","addr":"172.21.15.38:6804","nonce":13357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":13357},{"type":"v1","addr":"172.21.15.38:6811","nonce":13357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":13357},{"type":"v1","addr":"172.21.15.38:6822","nonce":13357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":13357},{"type":"v1","addr":"172.21.15.38:6817","nonce":13357}]},"public_addr":"172.21.15.38:6804/13357","cluster_addr":"172.21.15.38:6811/13357","heartbeat_back_addr":"172.21.15.38:6822/13357","heartbeat_front_addr":"172.21.15.38:6817/13357","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.332536+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.619276+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.340269+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:38:29.482 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-10-15T16:38:23.114481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-10-15T16:38:27.750568+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-10-15T16:38:29.483 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-10-15T16:38:29.777 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2020-10-15T16:38:29.791 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-10-15T16:38:30.085 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 8 2020-10-15T16:38:30.099 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-10-15T16:38:30.099 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-10-15T16:38:30.100 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-10-15T16:38:30.430 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:30.444 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":13171},{"type":"v1","addr":"172.21.15.38:6825","nonce":13171}]},"active_addr":"172.21.15.38:6825/13171","active_change":"2020-10-15T16:38:23.086228+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1421694333}]},{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":361157173}]},{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":641485554}]}]}} 2020-10-15T16:38:30.446 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-10-15T16:38:30.446 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-10-15T16:38:30.447 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-10-15T16:38:30.737 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:30.738 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"fsid":"95640a16-2ac6-41cc-ab44-74701d09dc50","created":"2020-10-15T16:38:19.059495+0000","modified":"2020-10-15T16:38:30.159313+0000","last_up_change":"2020-10-15T16:38:24.098780+0000","last_in_change":"2020-10-15T16:38:24.098780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:38:23.114481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-15T16:38:27.750568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5b5187a2-4591-4988-b6dc-17bdd5679d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":13355},{"type":"v1","addr":"172.21.15.38:6806","nonce":13355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":13355},{"type":"v1","addr":"172.21.15.38:6812","nonce":13355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6821","nonce":13355},{"type":"v1","addr":"172.21.15.38:6823","nonce":13355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":13355},{"type":"v1","addr":"172.21.15.38:6818","nonce":13355}]},"public_addr":"172.21.15.38:6806/13355","cluster_addr":"172.21.15.38:6812/13355","heartbeat_back_addr":"172.21.15.38:6823/13355","heartbeat_front_addr":"172.21.15.38:6818/13355","state":["exists","up"]},{"osd":1,"uuid":"809c9565-3e26-4295-b0d3-2b0414d350f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":13356},{"type":"v1","addr":"172.21.15.38:6801","nonce":13356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":13356},{"type":"v1","addr":"172.21.15.38:6807","nonce":13356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":13356},{"type":"v1","addr":"172.21.15.38:6819","nonce":13356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":13356},{"type":"v1","addr":"172.21.15.38:6813","nonce":13356}]},"public_addr":"172.21.15.38:6801/13356","cluster_addr":"172.21.15.38:6807/13356","heartbeat_back_addr":"172.21.15.38:6819/13356","heartbeat_front_addr":"172.21.15.38:6813/13356","state":["exists","up"]},{"osd":2,"uuid":"9433d174-9f6b-44d6-817a-8c8465a2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":13357},{"type":"v1","addr":"172.21.15.38:6804","nonce":13357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":13357},{"type":"v1","addr":"172.21.15.38:6811","nonce":13357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":13357},{"type":"v1","addr":"172.21.15.38:6822","nonce":13357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":13357},{"type":"v1","addr":"172.21.15.38:6817","nonce":13357}]},"public_addr":"172.21.15.38:6804/13357","cluster_addr":"172.21.15.38:6811/13357","heartbeat_back_addr":"172.21.15.38:6822/13357","heartbeat_front_addr":"172.21.15.38:6817/13357","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.332536+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.619276+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.340269+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:38:30.752 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-10-15T16:38:30.752 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-10-15T16:38:31.041 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:31.041 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"fsid":"95640a16-2ac6-41cc-ab44-74701d09dc50","created":"2020-10-15T16:38:19.059495+0000","modified":"2020-10-15T16:38:30.159313+0000","last_up_change":"2020-10-15T16:38:24.098780+0000","last_in_change":"2020-10-15T16:38:24.098780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-15T16:38:23.114481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-15T16:38:27.750568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5b5187a2-4591-4988-b6dc-17bdd5679d71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":13355},{"type":"v1","addr":"172.21.15.38:6806","nonce":13355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":13355},{"type":"v1","addr":"172.21.15.38:6812","nonce":13355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6821","nonce":13355},{"type":"v1","addr":"172.21.15.38:6823","nonce":13355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":13355},{"type":"v1","addr":"172.21.15.38:6818","nonce":13355}]},"public_addr":"172.21.15.38:6806/13355","cluster_addr":"172.21.15.38:6812/13355","heartbeat_back_addr":"172.21.15.38:6823/13355","heartbeat_front_addr":"172.21.15.38:6818/13355","state":["exists","up"]},{"osd":1,"uuid":"809c9565-3e26-4295-b0d3-2b0414d350f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":13356},{"type":"v1","addr":"172.21.15.38:6801","nonce":13356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":13356},{"type":"v1","addr":"172.21.15.38:6807","nonce":13356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":13356},{"type":"v1","addr":"172.21.15.38:6819","nonce":13356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":13356},{"type":"v1","addr":"172.21.15.38:6813","nonce":13356}]},"public_addr":"172.21.15.38:6801/13356","cluster_addr":"172.21.15.38:6807/13356","heartbeat_back_addr":"172.21.15.38:6819/13356","heartbeat_front_addr":"172.21.15.38:6813/13356","state":["exists","up"]},{"osd":2,"uuid":"9433d174-9f6b-44d6-817a-8c8465a2a0d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":13357},{"type":"v1","addr":"172.21.15.38:6804","nonce":13357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":13357},{"type":"v1","addr":"172.21.15.38:6811","nonce":13357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":13357},{"type":"v1","addr":"172.21.15.38:6822","nonce":13357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":13357},{"type":"v1","addr":"172.21.15.38:6817","nonce":13357}]},"public_addr":"172.21.15.38:6804/13357","cluster_addr":"172.21.15.38:6811/13357","heartbeat_back_addr":"172.21.15.38:6822/13357","heartbeat_front_addr":"172.21.15.38:6817/13357","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.332536+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.619276+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-15T16:38:22.340269+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-15T16:38:31.056 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-10-15T16:38:31.219 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2020-10-15T16:38:31.220 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-10-15T16:38:31.376 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2020-10-15T16:38:31.377 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-10-15T16:38:31.546 INFO:teuthology.orchestra.run.smithi038.stdout:34359738372 2020-10-15T16:38:31.547 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-10-15T16:38:31.840 INFO:teuthology.orchestra.run.smithi038.stdout:34359738370 2020-10-15T16:38:31.854 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-10-15T16:38:32.855 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-10-15T16:38:33.164 INFO:teuthology.orchestra.run.smithi038.stdout:34359738370 2020-10-15T16:38:33.179 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-10-15T16:38:34.180 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-10-15T16:38:34.475 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2020-10-15T16:38:34.490 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-10-15T16:38:34.490 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-10-15T16:38:34.783 INFO:teuthology.orchestra.run.smithi038.stdout:34359738371 2020-10-15T16:38:34.797 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-10-15T16:38:34.797 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-10-15T16:38:35.088 INFO:teuthology.orchestra.run.smithi038.stdout:34359738372 2020-10-15T16:38:35.102 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-10-15T16:38:35.103 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-10-15T16:38:35.103 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-10-15T16:38:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:35.389 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2020-10-15T16:38:35.401 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-10-15T16:38:35.103851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14760,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558040,"statfs":{"total":322122547200,"available":322107432960,"internally_reserved":0,"allocated":344064,"data_stored":98208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172240+0000","last_change":"2020-10-15T16:38:29.167653+0000","last_active":"2020-10-15T16:38:29.172240+0000","last_peered":"2020-10-15T16:38:29.172240+0000","last_clean":"2020-10-15T16:38:29.172240+0000","last_became_active":"2020-10-15T16:38:29.167232+0000","last_became_peered":"2020-10-15T16:38:29.167232+0000","last_unstale":"2020-10-15T16:38:29.172240+0000","last_undegraded":"2020-10-15T16:38:29.172240+0000","last_fullsized":"2020-10-15T16:38:29.172240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172294+0000","last_change":"2020-10-15T16:38:29.168495+0000","last_active":"2020-10-15T16:38:29.172294+0000","last_peered":"2020-10-15T16:38:29.172294+0000","last_clean":"2020-10-15T16:38:29.172294+0000","last_became_active":"2020-10-15T16:38:29.167780+0000","last_became_peered":"2020-10-15T16:38:29.167780+0000","last_unstale":"2020-10-15T16:38:29.172294+0000","last_undegraded":"2020-10-15T16:38:29.172294+0000","last_fullsized":"2020-10-15T16:38:29.172294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172724+0000","last_change":"2020-10-15T16:38:29.170762+0000","last_active":"2020-10-15T16:38:29.172724+0000","last_peered":"2020-10-15T16:38:29.172724+0000","last_clean":"2020-10-15T16:38:29.172724+0000","last_became_active":"2020-10-15T16:38:29.170432+0000","last_became_peered":"2020-10-15T16:38:29.170432+0000","last_unstale":"2020-10-15T16:38:29.172724+0000","last_undegraded":"2020-10-15T16:38:29.172724+0000","last_fullsized":"2020-10-15T16:38:29.172724+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.167809+0000","last_change":"2020-10-15T16:38:29.165523+0000","last_active":"2020-10-15T16:38:29.167809+0000","last_peered":"2020-10-15T16:38:29.167809+0000","last_clean":"2020-10-15T16:38:29.167809+0000","last_became_active":"2020-10-15T16:38:29.164581+0000","last_became_peered":"2020-10-15T16:38:29.164581+0000","last_unstale":"2020-10-15T16:38:29.167809+0000","last_undegraded":"2020-10-15T16:38:29.167809+0000","last_fullsized":"2020-10-15T16:38:29.167809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.161941+0000","last_change":"2020-10-15T16:38:25.379048+0000","last_active":"2020-10-15T16:38:29.161941+0000","last_peered":"2020-10-15T16:38:29.161941+0000","last_clean":"2020-10-15T16:38:29.161941+0000","last_became_active":"2020-10-15T16:38:25.377713+0000","last_became_peered":"2020-10-15T16:38:25.377713+0000","last_unstale":"2020-10-15T16:38:29.161941+0000","last_undegraded":"2020-10-15T16:38:29.161941+0000","last_fullsized":"2020-10-15T16:38:29.161941+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:24.098780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:24.098780+0000","last_clean_scrub_stamp":"2020-10-15T16:38:24.098780+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172312+0000","last_change":"2020-10-15T16:38:29.168881+0000","last_active":"2020-10-15T16:38:29.172312+0000","last_peered":"2020-10-15T16:38:29.172312+0000","last_clean":"2020-10-15T16:38:29.172312+0000","last_became_active":"2020-10-15T16:38:29.168527+0000","last_became_peered":"2020-10-15T16:38:29.168527+0000","last_unstale":"2020-10-15T16:38:29.172312+0000","last_undegraded":"2020-10-15T16:38:29.172312+0000","last_fullsized":"2020-10-15T16:38:29.172312+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.169920+0000","last_change":"2020-10-15T16:38:29.166315+0000","last_active":"2020-10-15T16:38:29.169920+0000","last_peered":"2020-10-15T16:38:29.169920+0000","last_clean":"2020-10-15T16:38:29.169920+0000","last_became_active":"2020-10-15T16:38:29.165705+0000","last_became_peered":"2020-10-15T16:38:29.165705+0000","last_unstale":"2020-10-15T16:38:29.169920+0000","last_undegraded":"2020-10-15T16:38:29.169920+0000","last_fullsized":"2020-10-15T16:38:29.169920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.174716+0000","last_change":"2020-10-15T16:38:29.173414+0000","last_active":"2020-10-15T16:38:29.174716+0000","last_peered":"2020-10-15T16:38:29.174716+0000","last_clean":"2020-10-15T16:38:29.174716+0000","last_became_active":"2020-10-15T16:38:29.172966+0000","last_became_peered":"2020-10-15T16:38:29.172966+0000","last_unstale":"2020-10-15T16:38:29.174716+0000","last_undegraded":"2020-10-15T16:38:29.174716+0000","last_fullsized":"2020-10-15T16:38:29.174716+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172718+0000","last_change":"2020-10-15T16:38:29.171289+0000","last_active":"2020-10-15T16:38:29.172718+0000","last_peered":"2020-10-15T16:38:29.172718+0000","last_clean":"2020-10-15T16:38:29.172718+0000","last_became_active":"2020-10-15T16:38:29.170823+0000","last_became_peered":"2020-10-15T16:38:29.170823+0000","last_unstale":"2020-10-15T16:38:29.172718+0000","last_undegraded":"2020-10-15T16:38:29.172718+0000","last_fullsized":"2020-10-15T16:38:29.172718+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.704}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]}]}]}} 2020-10-15T16:38:35.402 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-10-15T16:38:35.687 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:35.690 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2020-10-15T16:38:35.701 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-10-15T16:38:35.103851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14760,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558040,"statfs":{"total":322122547200,"available":322107432960,"internally_reserved":0,"allocated":344064,"data_stored":98208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.939623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172240+0000","last_change":"2020-10-15T16:38:29.167653+0000","last_active":"2020-10-15T16:38:29.172240+0000","last_peered":"2020-10-15T16:38:29.172240+0000","last_clean":"2020-10-15T16:38:29.172240+0000","last_became_active":"2020-10-15T16:38:29.167232+0000","last_became_peered":"2020-10-15T16:38:29.167232+0000","last_unstale":"2020-10-15T16:38:29.172240+0000","last_undegraded":"2020-10-15T16:38:29.172240+0000","last_fullsized":"2020-10-15T16:38:29.172240+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172294+0000","last_change":"2020-10-15T16:38:29.168495+0000","last_active":"2020-10-15T16:38:29.172294+0000","last_peered":"2020-10-15T16:38:29.172294+0000","last_clean":"2020-10-15T16:38:29.172294+0000","last_became_active":"2020-10-15T16:38:29.167780+0000","last_became_peered":"2020-10-15T16:38:29.167780+0000","last_unstale":"2020-10-15T16:38:29.172294+0000","last_undegraded":"2020-10-15T16:38:29.172294+0000","last_fullsized":"2020-10-15T16:38:29.172294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172724+0000","last_change":"2020-10-15T16:38:29.170762+0000","last_active":"2020-10-15T16:38:29.172724+0000","last_peered":"2020-10-15T16:38:29.172724+0000","last_clean":"2020-10-15T16:38:29.172724+0000","last_became_active":"2020-10-15T16:38:29.170432+0000","last_became_peered":"2020-10-15T16:38:29.170432+0000","last_unstale":"2020-10-15T16:38:29.172724+0000","last_undegraded":"2020-10-15T16:38:29.172724+0000","last_fullsized":"2020-10-15T16:38:29.172724+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.167809+0000","last_change":"2020-10-15T16:38:29.165523+0000","last_active":"2020-10-15T16:38:29.167809+0000","last_peered":"2020-10-15T16:38:29.167809+0000","last_clean":"2020-10-15T16:38:29.167809+0000","last_became_active":"2020-10-15T16:38:29.164581+0000","last_became_peered":"2020-10-15T16:38:29.164581+0000","last_unstale":"2020-10-15T16:38:29.167809+0000","last_undegraded":"2020-10-15T16:38:29.167809+0000","last_fullsized":"2020-10-15T16:38:29.167809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.161941+0000","last_change":"2020-10-15T16:38:25.379048+0000","last_active":"2020-10-15T16:38:29.161941+0000","last_peered":"2020-10-15T16:38:29.161941+0000","last_clean":"2020-10-15T16:38:29.161941+0000","last_became_active":"2020-10-15T16:38:25.377713+0000","last_became_peered":"2020-10-15T16:38:25.377713+0000","last_unstale":"2020-10-15T16:38:29.161941+0000","last_undegraded":"2020-10-15T16:38:29.161941+0000","last_fullsized":"2020-10-15T16:38:29.161941+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:24.098780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:24.098780+0000","last_clean_scrub_stamp":"2020-10-15T16:38:24.098780+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172312+0000","last_change":"2020-10-15T16:38:29.168881+0000","last_active":"2020-10-15T16:38:29.172312+0000","last_peered":"2020-10-15T16:38:29.172312+0000","last_clean":"2020-10-15T16:38:29.172312+0000","last_became_active":"2020-10-15T16:38:29.168527+0000","last_became_peered":"2020-10-15T16:38:29.168527+0000","last_unstale":"2020-10-15T16:38:29.172312+0000","last_undegraded":"2020-10-15T16:38:29.172312+0000","last_fullsized":"2020-10-15T16:38:29.172312+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.169920+0000","last_change":"2020-10-15T16:38:29.166315+0000","last_active":"2020-10-15T16:38:29.169920+0000","last_peered":"2020-10-15T16:38:29.169920+0000","last_clean":"2020-10-15T16:38:29.169920+0000","last_became_active":"2020-10-15T16:38:29.165705+0000","last_became_peered":"2020-10-15T16:38:29.165705+0000","last_unstale":"2020-10-15T16:38:29.169920+0000","last_undegraded":"2020-10-15T16:38:29.169920+0000","last_fullsized":"2020-10-15T16:38:29.169920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.174716+0000","last_change":"2020-10-15T16:38:29.173414+0000","last_active":"2020-10-15T16:38:29.174716+0000","last_peered":"2020-10-15T16:38:29.174716+0000","last_clean":"2020-10-15T16:38:29.174716+0000","last_became_active":"2020-10-15T16:38:29.172966+0000","last_became_peered":"2020-10-15T16:38:29.172966+0000","last_unstale":"2020-10-15T16:38:29.174716+0000","last_undegraded":"2020-10-15T16:38:29.174716+0000","last_fullsized":"2020-10-15T16:38:29.174716+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-15T16:38:29.172718+0000","last_change":"2020-10-15T16:38:29.171289+0000","last_active":"2020-10-15T16:38:29.172718+0000","last_peered":"2020-10-15T16:38:29.172718+0000","last_clean":"2020-10-15T16:38:29.172718+0000","last_became_active":"2020-10-15T16:38:29.170823+0000","last_became_peered":"2020-10-15T16:38:29.170823+0000","last_unstale":"2020-10-15T16:38:29.172718+0000","last_undegraded":"2020-10-15T16:38:29.172718+0000","last_fullsized":"2020-10-15T16:38:29.172718+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-15T16:38:28.137485+0000","last_clean_scrub_stamp":"2020-10-15T16:38:28.137485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.704}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]}]}]}} 2020-10-15T16:38:35.702 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-10-15T16:38:35.702 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-10-15T16:38:35.703 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-10-15T16:38:35.703 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-10-15T16:38:36.039 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T16:38:36.053 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-10-15T16:38:36.053 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-10-15T16:38:36.054 INFO:teuthology.run_tasks:Running task rgw... 2020-10-15T16:38:36.072 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-10-15T16:38:36.073 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-10-15T16:38:36.073 INFO:tasks.rgw:Creating data pools 2020-10-15T16:38:36.073 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-10-15T16:38:36.074 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-10-15T16:38:36.961 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'default.rgw.buckets.data' created 2020-10-15T16:38:36.983 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-10-15T16:38:37.991 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-15T16:38:38.004 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-10-15T16:38:38.983 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'default.rgw.buckets.index' created 2020-10-15T16:38:39.002 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-10-15T16:38:40.002 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-15T16:38:40.023 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-10-15T16:38:40.024 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-10-15T16:38:40.304 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'default.rgw.buckets.data' already exists 2020-10-15T16:38:40.315 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-10-15T16:38:40.983 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-15T16:38:40.998 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-10-15T16:38:41.300 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'default.rgw.buckets.index' already exists 2020-10-15T16:38:41.311 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-10-15T16:38:41.999 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-15T16:38:42.011 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-10-15T16:38:42.011 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-10-15T16:38:43.080 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'default.rgw.buckets.data' created 2020-10-15T16:38:43.102 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-10-15T16:38:44.106 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-15T16:38:44.127 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-10-15T16:38:45.091 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'default.rgw.buckets.index' created 2020-10-15T16:38:45.115 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-10-15T16:38:46.117 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-15T16:38:46.140 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-10-15T16:38:46.141 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-10-15T16:38:46.438 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'default.rgw.buckets.data' already exists 2020-10-15T16:38:46.449 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-10-15T16:38:47.110 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-15T16:38:47.124 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-10-15T16:38:47.421 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'default.rgw.buckets.index' already exists 2020-10-15T16:38:47.432 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-10-15T16:38:48.123 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-15T16:38:48.137 DEBUG:tasks.rgw:Pools created 2020-10-15T16:38:48.138 INFO:tasks.rgw:Configuring compression type = random 2020-10-15T16:38:48.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-10-15T16:38:48.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-10-15T16:38:48.139 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-10-15T16:38:48.176 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:38:48.177 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:38:48.188 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:48.190+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.953 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.953+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.954 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.953+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.954 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.953+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.954 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.953+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.955 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.953+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.955 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.953+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.965 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.965+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.981 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.982 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.982 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.982 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.982 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.983 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.983 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 10 cannot find current period zonegroup using local zonegroup 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 10 Creating default zonegroup 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 20 get_system_obj_state: rctx=0x7fff838314b0 obj=:.pools.avail state=0x5629ed5f22e0 s->prefetch_data=0 2020-10-15T16:38:49.984 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:49.981+0000 7fb3d81900c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-10-15T16:38:50.000 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.001+0000 7fb3d81900c0 10 WARNING: store->list_zones() returned r=-2 2020-10-15T16:38:50.000 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.001+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.001 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.001+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:50.006 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.005+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.007 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.005+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:50.007 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.005+0000 7fb3d81900c0 10 could not read realm id: (2) No such file or directory 2020-10-15T16:38:50.008 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.005+0000 7fb3d81900c0 10 WARNING: failed to set zone as default, r=-22 2020-10-15T16:38:50.008 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.005+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.008 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.009+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:50.013 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.014 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-15T16:38:50.014 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.015 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:38:50.015 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 zonegroup default 2020-10-15T16:38:50.015 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.016 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:50.016 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 10 Cannot find current period zone using local zone 2020-10-15T16:38:50.016 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 10 Using default name default 2020-10-15T16:38:50.016 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.013+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.017 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.017+0000 7fb3d81900c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:50.017 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.017+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:50.018 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.017+0000 7fb3d81900c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-15T16:38:50.018 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.017+0000 7fb3d81900c0 20 zone default found 2020-10-15T16:38:50.018 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:50.017+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:51.979 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:51.976+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:51.980 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:51.976+0000 7fb3d81900c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:51.980 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:51.976+0000 7fb3d81900c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:51.980 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:51.976+0000 7fb3d81900c0 20 started sync module instance, tier type = 2020-10-15T16:38:51.981 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:51.976+0000 7fb3d81900c0 20 started zone id=3f7838e9-ec10-4ba1-aed3-c798f1920a33 (name=default) with tier type = 2020-10-15T16:38:54.046 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=0 2020-10-15T16:38:54.046 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=1 2020-10-15T16:38:54.046 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=2 2020-10-15T16:38:54.047 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=3 2020-10-15T16:38:54.047 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=4 2020-10-15T16:38:54.047 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=5 2020-10-15T16:38:54.047 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.043+0000 7fb3d81900c0 20 add_watcher() i=6 2020-10-15T16:38:54.047 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.047+0000 7fb3d81900c0 20 add_watcher() i=7 2020-10-15T16:38:54.048 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.047+0000 7fb3d81900c0 2 all 8 watchers are set, enabling cache 2020-10-15T16:38:54.075 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.075+0000 7fb3957e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:38:54.075 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.075+0000 7fb3d81900c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:38:54.075 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:54.075+0000 7fb3d81900c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:38:58.013 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.009+0000 7fb3d81900c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-10-15T16:38:58.122 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.117+0000 7fb2f7fff700 20 reqs_thread_entry: start 2020-10-15T16:38:58.123 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.117+0000 7fb3d81900c0 20 init_complete bucket index max shards: 11 2020-10-15T16:38:58.123 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.117+0000 7fb2f5ffb700 20 reqs_thread_entry: start 2020-10-15T16:38:58.123 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.117+0000 7fb3d81900c0 10 Started notification manager with: 1 workers 2020-10-15T16:38:58.123 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.117+0000 7fb2e3fff700 20 INFO: next queues processing will happen at: Thu Oct 15 16:39:28 2020 2020-10-15T16:38:58.124 INFO:teuthology.orchestra.run.smithi183.stderr: 2020-10-15T16:38:58.124 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.117+0000 7fb2e37fe700 20 reqs_thread_entry: start 2020-10-15T16:38:58.154 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:38:58.154 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.149+0000 7fb3d81900c0 20 remove_watcher() i=0 2020-10-15T16:38:58.154 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.149+0000 7fb3d81900c0 2 removed watcher, disabling cache 2020-10-15T16:38:58.157 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.153+0000 7fb3d81900c0 20 remove_watcher() i=1 2020-10-15T16:38:58.159 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.157+0000 7fb3d81900c0 20 remove_watcher() i=2 2020-10-15T16:38:58.162 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.157+0000 7fb3d81900c0 20 remove_watcher() i=3 2020-10-15T16:38:58.165 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.161+0000 7fb3d81900c0 20 remove_watcher() i=4 2020-10-15T16:38:58.168 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.165+0000 7fb3d81900c0 20 remove_watcher() i=5 2020-10-15T16:38:58.171 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.165+0000 7fb3d81900c0 20 remove_watcher() i=6 2020-10-15T16:38:58.173 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.169+0000 7fb3d81900c0 20 remove_watcher() i=7 2020-10-15T16:38:58.175 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.169+0000 7fb3957e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:38:58.196 DEBUG:tasks.util.rgw: json result: [] 2020-10-15T16:38:58.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:38:58.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:38:58.197 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-15T16:38:58.280 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:38:58.280 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:38:58.293 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.289+0000 7f339b6e80c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:38:58.294 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.289+0000 7f339b6e80c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:38:58.299 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.293+0000 7f32cbfff700 20 reqs_thread_entry: start 2020-10-15T16:38:58.299 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.293+0000 7f339b6e80c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.300 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.300 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.302 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-15T16:38:58.302 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.303 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.303 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.303 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.304 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.297+0000 7f339b6e80c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.304 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.301+0000 7f339b6e80c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:38:58.312 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:38:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "3f7838e9-ec10-4ba1-aed3-c798f1920a33", 2020-10-15T16:38:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default", 2020-10-15T16:38:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "default.rgw.meta:root", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "default.rgw.control", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "default.rgw.log", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-15T16:38:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "default.rgw.otp", 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:38:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "", 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "" 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:38:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "compression_type": "random" 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:38:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "", 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-15T16:38:58.317 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:38:58.318 DEBUG:tasks.util.rgw: json result: {'id': '3f7838e9-ec10-4ba1-aed3-c798f1920a33', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-15T16:38:58.319 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-10-15T16:38:58.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-10-15T16:38:58.319 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-10-15T16:38:58.395 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:38:58.395 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:38:58.404 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.401+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.407 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.401+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.407 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.401+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.407 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.405+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.408 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.405+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.408 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.405+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.408 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.405+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.409 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.405+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.436 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 RGWRados::pool_iterate: got zonegroup_info.56f834a6-e3b2-426c-87c1-1657a291ddc5 2020-10-15T16:38:58.437 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 RGWRados::pool_iterate: got zone_info.3f7838e9-ec10-4ba1-aed3-c798f1920a33 2020-10-15T16:38:58.437 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 RGWRados::pool_iterate: got zone_names.default 2020-10-15T16:38:58.437 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-10-15T16:38:58.437 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.437 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.438 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.439 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-15T16:38:58.439 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.433+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.440 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.440 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.441 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:38:58.441 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.442 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.442 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 10 cannot find current period zonegroup using local zonegroup 2020-10-15T16:38:58.442 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.443 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.443 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.437+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.444 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.444 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.445 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:38:58.445 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 zonegroup default 2020-10-15T16:38:58.445 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.445 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.446 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 10 Cannot find current period zone using local zone 2020-10-15T16:38:58.446 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.447 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.447 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.447 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.441+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.448 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.448 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-15T16:38:58.448 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 zone default found 2020-10-15T16:38:58.449 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.449 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.449 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.450 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.450 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 started sync module instance, tier type = 2020-10-15T16:38:58.451 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.445+0000 7f801544a0c0 20 started zone id=3f7838e9-ec10-4ba1-aed3-c798f1920a33 (name=default) with tier type = 2020-10-15T16:38:58.487 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=0 2020-10-15T16:38:58.487 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=1 2020-10-15T16:38:58.487 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=2 2020-10-15T16:38:58.488 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=3 2020-10-15T16:38:58.488 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=4 2020-10-15T16:38:58.488 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=5 2020-10-15T16:38:58.488 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.481+0000 7f801544a0c0 20 add_watcher() i=6 2020-10-15T16:38:58.489 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.485+0000 7f801544a0c0 20 add_watcher() i=7 2020-10-15T16:38:58.489 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.485+0000 7f801544a0c0 2 all 8 watchers are set, enabling cache 2020-10-15T16:38:58.512 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.509+0000 7f7fd27e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:38:58.512 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.509+0000 7f801544a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:38:58.512 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.509+0000 7f801544a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:38:58.615 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.609+0000 7f7f41ffb700 20 reqs_thread_entry: start 2020-10-15T16:38:58.617 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.613+0000 7f801544a0c0 20 init_complete bucket index max shards: 11 2020-10-15T16:38:58.617 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.613+0000 7f7f3affd700 20 reqs_thread_entry: start 2020-10-15T16:38:58.617 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.613+0000 7f801544a0c0 10 Started notification manager with: 1 workers 2020-10-15T16:38:58.617 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.613+0000 7f7f397fa700 20 INFO: next queues processing will happen at: Thu Oct 15 16:39:29 2020 2020-10-15T16:38:58.617 INFO:teuthology.orchestra.run.smithi183.stderr: 2020-10-15T16:38:58.617 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.613+0000 7f7f38ff9700 20 reqs_thread_entry: start 2020-10-15T16:38:58.646 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:38:58.647 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.641+0000 7f801544a0c0 20 remove_watcher() i=0 2020-10-15T16:38:58.647 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.641+0000 7f801544a0c0 2 removed watcher, disabling cache 2020-10-15T16:38:58.649 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.645+0000 7f801544a0c0 20 remove_watcher() i=1 2020-10-15T16:38:58.653 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.649+0000 7f801544a0c0 20 remove_watcher() i=2 2020-10-15T16:38:58.654 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.649+0000 7f801544a0c0 20 remove_watcher() i=3 2020-10-15T16:38:58.656 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.653+0000 7f801544a0c0 20 remove_watcher() i=4 2020-10-15T16:38:58.659 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.653+0000 7f801544a0c0 20 remove_watcher() i=5 2020-10-15T16:38:58.661 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.657+0000 7f801544a0c0 20 remove_watcher() i=6 2020-10-15T16:38:58.664 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.661+0000 7f801544a0c0 20 remove_watcher() i=7 2020-10-15T16:38:58.665 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.661+0000 7f7fd27e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:38:58.672 DEBUG:tasks.util.rgw: json result: [] 2020-10-15T16:38:58.672 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:38:58.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:38:58.673 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-15T16:38:58.755 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:38:58.756 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:38:58.769 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.765+0000 7fdd08b1b0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:38:58.770 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.765+0000 7fdd08b1b0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:38:58.777 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.773+0000 7fdc657fa700 20 reqs_thread_entry: start 2020-10-15T16:38:58.777 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.773+0000 7fdd08b1b0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.778 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.773+0000 7fdd08b1b0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.778 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.773+0000 7fdd08b1b0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.780 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-15T16:38:58.780 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.781 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:38:58.781 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.782 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:38:58.782 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados->read ofs=0 len=0 2020-10-15T16:38:58.783 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:38:58.777+0000 7fdd08b1b0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:38:58.791 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:38:58.791 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "3f7838e9-ec10-4ba1-aed3-c798f1920a33", 2020-10-15T16:38:58.792 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default", 2020-10-15T16:38:58.792 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "default.rgw.meta:root", 2020-10-15T16:38:58.792 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "default.rgw.control", 2020-10-15T16:38:58.792 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-15T16:38:58.792 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-15T16:38:58.792 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "default.rgw.log", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-15T16:38:58.793 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "default.rgw.otp", 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "", 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "" 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:38:58.794 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:38:58.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:38:58.795 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:38:58.795 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:38:58.795 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-15T16:38:58.795 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:38:58.795 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:38:58.796 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-15T16:38:58.796 INFO:teuthology.orchestra.run.smithi183.stdout: "compression_type": "random" 2020-10-15T16:38:58.796 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:38:58.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:38:58.797 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-15T16:38:58.798 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:38:58.798 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:38:58.798 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:38:58.798 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:38:58.798 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "", 2020-10-15T16:38:58.798 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-15T16:38:58.799 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:38:58.799 DEBUG:tasks.util.rgw: json result: {'id': '3f7838e9-ec10-4ba1-aed3-c798f1920a33', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-15T16:38:58.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-10-15T16:38:58.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-10-15T16:38:58.801 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-10-15T16:38:58.837 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:38:58.837 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:38:58.846 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:38:58.842+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.105 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.102+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.106 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.102+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.106 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.102+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.106 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.102+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.108 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.106+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.108 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.106+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.109 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.106+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.132 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.132 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.133 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.133 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.133 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.133 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.133 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 10 cannot find current period zonegroup using local zonegroup 2020-10-15T16:39:00.133 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.134 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.134 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.134 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.135 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 10 Creating default zonegroup 2020-10-15T16:39:00.135 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 20 get_system_obj_state: rctx=0x7ffdd9dd1400 obj=:.pools.avail state=0x562bf48872e0 s->prefetch_data=0 2020-10-15T16:39:00.135 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.130+0000 7f06c42660c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-10-15T16:39:00.149 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.146+0000 7f06c42660c0 10 WARNING: store->list_zones() returned r=-2 2020-10-15T16:39:00.149 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.146+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.149 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.146+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.154 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.150+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.154 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.150+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.154 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.150+0000 7f06c42660c0 10 could not read realm id: (2) No such file or directory 2020-10-15T16:39:00.154 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.150+0000 7f06c42660c0 10 WARNING: failed to set zone as default, r=-22 2020-10-15T16:39:00.155 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.150+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.155 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.154+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.159 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.158+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.160 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.158+0000 7f06c42660c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-15T16:39:00.160 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.158+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.163 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:39:00.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 zonegroup default 2020-10-15T16:39:00.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:00.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 10 Cannot find current period zone using local zone 2020-10-15T16:39:00.165 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 10 Using default name default 2020-10-15T16:39:00.165 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.165 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:00.165 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:00.166 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-15T16:39:00.166 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 zone default found 2020-10-15T16:39:00.166 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:00.162+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:02.122 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:02.117+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:02.122 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:02.117+0000 7f06c42660c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:02.126 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:02.121+0000 7f06c42660c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:02.127 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:02.121+0000 7f06c42660c0 20 started sync module instance, tier type = 2020-10-15T16:39:02.127 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:02.121+0000 7f06c42660c0 20 started zone id=8631ea27-0f9c-453e-aa1c-bc19d7d5eac9 (name=default) with tier type = 2020-10-15T16:39:04.163 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=0 2020-10-15T16:39:04.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=1 2020-10-15T16:39:04.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=2 2020-10-15T16:39:04.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=3 2020-10-15T16:39:04.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=4 2020-10-15T16:39:04.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=5 2020-10-15T16:39:04.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=6 2020-10-15T16:39:04.165 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 20 add_watcher() i=7 2020-10-15T16:39:04.165 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.161+0000 7f06c42660c0 2 all 8 watchers are set, enabling cache 2020-10-15T16:39:04.191 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.185+0000 7f0680ff1700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:39:04.191 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.189+0000 7f06c42660c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:39:04.191 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:04.189+0000 7f06c42660c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:39:08.150 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.145+0000 7f06c42660c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-10-15T16:39:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.265+0000 7f05e37fe700 20 reqs_thread_entry: start 2020-10-15T16:39:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.265+0000 7f06c42660c0 20 init_complete bucket index max shards: 11 2020-10-15T16:39:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.265+0000 7f05e17fa700 20 reqs_thread_entry: start 2020-10-15T16:39:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.265+0000 7f06c42660c0 10 Started notification manager with: 1 workers 2020-10-15T16:39:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.265+0000 7f05cf7fe700 20 INFO: next queues processing will happen at: Thu Oct 15 16:39:38 2020 2020-10-15T16:39:08.270 INFO:teuthology.orchestra.run.smithi038.stderr: 2020-10-15T16:39:08.270 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.265+0000 7f05ceffd700 20 reqs_thread_entry: start 2020-10-15T16:39:08.299 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:39:08.299 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.297+0000 7f06c42660c0 20 remove_watcher() i=0 2020-10-15T16:39:08.300 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.297+0000 7f06c42660c0 2 removed watcher, disabling cache 2020-10-15T16:39:08.302 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.297+0000 7f06c42660c0 20 remove_watcher() i=1 2020-10-15T16:39:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.301+0000 7f06c42660c0 20 remove_watcher() i=2 2020-10-15T16:39:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.305+0000 7f06c42660c0 20 remove_watcher() i=3 2020-10-15T16:39:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.305+0000 7f06c42660c0 20 remove_watcher() i=4 2020-10-15T16:39:08.313 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.309+0000 7f06c42660c0 20 remove_watcher() i=5 2020-10-15T16:39:08.316 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.313+0000 7f06c42660c0 20 remove_watcher() i=6 2020-10-15T16:39:08.319 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.317+0000 7f06c42660c0 20 remove_watcher() i=7 2020-10-15T16:39:08.321 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.317+0000 7f0680ff1700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:39:08.332 DEBUG:tasks.util.rgw: json result: [] 2020-10-15T16:39:08.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:39:08.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:39:08.333 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-15T16:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:08.425 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.421+0000 7efc14d650c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:39:08.426 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.421+0000 7efc14d650c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:39:08.435 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.429+0000 7efb4a7fc700 20 reqs_thread_entry: start 2020-10-15T16:39:08.435 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.429+0000 7efc14d650c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.436 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.433+0000 7efc14d650c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.437 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.433+0000 7efc14d650c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.438 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.433+0000 7efc14d650c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-15T16:39:08.438 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.433+0000 7efc14d650c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.439 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.437+0000 7efc14d650c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.439 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.437+0000 7efc14d650c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.440 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.437+0000 7efc14d650c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.440 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.437+0000 7efc14d650c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.441 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.437+0000 7efc14d650c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:39:08.454 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:39:08.455 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "8631ea27-0f9c-453e-aa1c-bc19d7d5eac9", 2020-10-15T16:39:08.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default", 2020-10-15T16:39:08.455 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "default.rgw.meta:root", 2020-10-15T16:39:08.455 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "default.rgw.control", 2020-10-15T16:39:08.455 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-15T16:39:08.455 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "default.rgw.log", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-15T16:39:08.456 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "default.rgw.otp", 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "", 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "" 2020-10-15T16:39:08.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:08.458 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:39:08.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:08.458 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:39:08.458 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:39:08.459 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-15T16:39:08.459 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:39:08.459 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:39:08.459 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-15T16:39:08.459 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:39:08.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:08.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:08.460 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-15T16:39:08.460 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:39:08.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:08.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:08.461 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:08.461 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "", 2020-10-15T16:39:08.461 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-15T16:39:08.461 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:39:08.462 DEBUG:tasks.util.rgw: json result: {'id': '8631ea27-0f9c-453e-aa1c-bc19d7d5eac9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-15T16:39:08.462 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-10-15T16:39:08.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-10-15T16:39:08.463 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-10-15T16:39:08.537 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:08.537 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:08.548 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.545+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.551 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.551 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.552 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.552 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.553 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.553 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.554 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.549+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 RGWRados::pool_iterate: got zone_info.8631ea27-0f9c-453e-aa1c-bc19d7d5eac9 2020-10-15T16:39:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 RGWRados::pool_iterate: got zone_names.default 2020-10-15T16:39:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-10-15T16:39:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 RGWRados::pool_iterate: got zonegroup_info.275d1506-55c3-4aac-aadc-c451889d45a0 2020-10-15T16:39:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-15T16:39:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.581+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:39:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 10 cannot find current period zonegroup using local zonegroup 2020-10-15T16:39:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.585+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.592 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:39:08.592 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 zonegroup default 2020-10-15T16:39:08.592 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.593 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.593 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 10 Cannot find current period zone using local zone 2020-10-15T16:39:08.593 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.594 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.594 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.589+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.595 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.595 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.596 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-15T16:39:08.596 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 zone default found 2020-10-15T16:39:08.596 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 started sync module instance, tier type = 2020-10-15T16:39:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.593+0000 7f1145ff80c0 20 started zone id=8631ea27-0f9c-453e-aa1c-bc19d7d5eac9 (name=default) with tier type = 2020-10-15T16:39:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=0 2020-10-15T16:39:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=1 2020-10-15T16:39:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=2 2020-10-15T16:39:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=3 2020-10-15T16:39:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=4 2020-10-15T16:39:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=5 2020-10-15T16:39:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=6 2020-10-15T16:39:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 20 add_watcher() i=7 2020-10-15T16:39:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.629+0000 7f1145ff80c0 2 all 8 watchers are set, enabling cache 2020-10-15T16:39:08.658 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.653+0000 7f11037de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:39:08.659 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.653+0000 7f1145ff80c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:39:08.659 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.653+0000 7f1145ff80c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:39:08.757 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.753+0000 7f10727fc700 20 reqs_thread_entry: start 2020-10-15T16:39:08.757 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.753+0000 7f106bfff700 20 reqs_thread_entry: start 2020-10-15T16:39:08.758 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.753+0000 7f1145ff80c0 20 init_complete bucket index max shards: 11 2020-10-15T16:39:08.758 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.753+0000 7f1145ff80c0 10 Started notification manager with: 1 workers 2020-10-15T16:39:08.758 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.753+0000 7f106a7fc700 20 INFO: next queues processing will happen at: Thu Oct 15 16:39:38 2020 2020-10-15T16:39:08.759 INFO:teuthology.orchestra.run.smithi038.stderr: 2020-10-15T16:39:08.759 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.753+0000 7f1069ffb700 20 reqs_thread_entry: start 2020-10-15T16:39:08.789 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:39:08.789 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.785+0000 7f1145ff80c0 20 remove_watcher() i=0 2020-10-15T16:39:08.790 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.785+0000 7f1145ff80c0 2 removed watcher, disabling cache 2020-10-15T16:39:08.792 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.789+0000 7f1145ff80c0 20 remove_watcher() i=1 2020-10-15T16:39:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.793+0000 7f1145ff80c0 20 remove_watcher() i=2 2020-10-15T16:39:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.797+0000 7f1145ff80c0 20 remove_watcher() i=3 2020-10-15T16:39:08.802 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.797+0000 7f1145ff80c0 20 remove_watcher() i=4 2020-10-15T16:39:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.801+0000 7f1145ff80c0 20 remove_watcher() i=5 2020-10-15T16:39:08.808 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.805+0000 7f1145ff80c0 20 remove_watcher() i=6 2020-10-15T16:39:08.811 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.809+0000 7f1145ff80c0 20 remove_watcher() i=7 2020-10-15T16:39:08.813 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.809+0000 7f11037de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-15T16:39:08.819 DEBUG:tasks.util.rgw: json result: [] 2020-10-15T16:39:08.820 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:39:08.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-15T16:39:08.820 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-15T16:39:08.904 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:08.905 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:08.918 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.913+0000 7f7efaa2e0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-15T16:39:08.918 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.913+0000 7f7efaa2e0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-15T16:39:08.924 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.921+0000 7f7efaa2e0c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.924 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.921+0000 7f7e2bfff700 20 reqs_thread_entry: start 2020-10-15T16:39:08.926 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.921+0000 7f7efaa2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.926 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.921+0000 7f7efaa2e0c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.927 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-15T16:39:08.927 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.934 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-15T16:39:08.934 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.934 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-15T16:39:08.934 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados->read ofs=0 len=0 2020-10-15T16:39:08.935 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:08.925+0000 7f7efaa2e0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-15T16:39:08.938 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:39:08.938 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "8631ea27-0f9c-453e-aa1c-bc19d7d5eac9", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "default.rgw.meta:root", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "default.rgw.control", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "default.rgw.log", 2020-10-15T16:39:08.939 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-15T16:39:08.940 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "default.rgw.otp", 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "", 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "" 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:39:08.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:08.942 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:39:08.942 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:39:08.942 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-15T16:39:08.942 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:39:08.942 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:39:08.942 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:08.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:08.944 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:08.944 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "", 2020-10-15T16:39:08.944 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-15T16:39:08.944 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:39:08.945 DEBUG:tasks.util.rgw: json result: {'id': '8631ea27-0f9c-453e-aa1c-bc19d7d5eac9', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-15T16:39:08.945 INFO:tasks.rgw:Starting rgw... 2020-10-15T16:39:08.946 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-10-15T16:39:08.946 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-15T16:39:08.946 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-10-15T16:39:08.947 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-10-15T16:39:08.949 INFO:tasks.rgw.c1.client.0:Started 2020-10-15T16:39:08.949 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-10-15T16:39:08.950 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-15T16:39:08.950 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-10-15T16:39:08.950 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-10-15T16:39:08.952 INFO:tasks.rgw.c1.client.1:Started 2020-10-15T16:39:08.953 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-10-15T16:39:08.953 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-15T16:39:08.953 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-10-15T16:39:08.954 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-10-15T16:39:08.988 INFO:tasks.rgw.c2.client.0:Started 2020-10-15T16:39:08.989 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-10-15T16:39:08.989 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-15T16:39:08.989 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-10-15T16:39:08.990 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-10-15T16:39:08.992 INFO:tasks.rgw.c2.client.1:Started 2020-10-15T16:39:08.992 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi183.front.sepia.ceph.com:8000/ 2020-10-15T16:39:08.992 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8000/ 2020-10-15T16:39:09.018 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:09.019 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:09.022 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi183.front.sepia.ceph.com port 8000: Connection refused 2020-10-15T16:39:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-15T16:39:10.025 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8000/ 2020-10-15T16:39:10.039 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:10.040 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:10.045 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-10-15T16:39:10.046 INFO:teuthology.orchestra.run.smithi183.stdout:anonymous 2020-10-15T16:39:10.046 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi183.front.sepia.ceph.com:8001/ 2020-10-15T16:39:10.047 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8001/ 2020-10-15T16:39:10.108 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:10.108 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:10.115 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 30571 2020-10-15T16:39:10.116 INFO:teuthology.orchestra.run.smithi183.stdout:anonymous 2020-10-15T16:39:10.116 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi038.front.sepia.ceph.com:8000/ 2020-10-15T16:39:10.117 INFO:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:8000/ 2020-10-15T16:39:10.149 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:10.150 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:10.155 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 16461 0 --:--:-- --:--:-- --:--:-- 42800 2020-10-15T16:39:10.156 INFO:teuthology.orchestra.run.smithi038.stdout:anonymous 2020-10-15T16:39:10.156 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi038.front.sepia.ceph.com:8001/ 2020-10-15T16:39:10.157 INFO:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:8001/ 2020-10-15T16:39:10.213 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:10.213 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:10.220 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-10-15T16:39:10.221 INFO:teuthology.orchestra.run.smithi038.stdout:anonymous 2020-10-15T16:39:10.222 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-10-15T16:39:10.594 INFO:tasks.rgw_multisite:creating realm.. 2020-10-15T16:39:10.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:10.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:10.596 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:10.632 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:10.632 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:10.681 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-realm", 2020-10-15T16:39:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "current_period": "dfa1fd57-6cc1-40c3-8be1-b0a367d9bbab", 2020-10-15T16:39:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 1 2020-10-15T16:39:10.682 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:10.683 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-10-15T16:39:10.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:10.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:10.684 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:10.766 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:10.767 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:10.785 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:39:10.781+0000 7fd9e777c0c0 -1 old format 2020-10-15T16:39:10.800 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:39:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:39:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:39:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:39:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:39:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:39:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "", 2020-10-15T16:39:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [], 2020-10-15T16:39:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [], 2020-10-15T16:39:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "", 2020-10-15T16:39:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:39:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:39:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:10.806 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:10.807 INFO:tasks.rgw_multisite:creating master zone.. 2020-10-15T16:39:10.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi183.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:10.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi183.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:10.809 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --endpoints http://smithi183.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:10.898 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:10.898 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:10.973 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:10.973 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:10.973 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:39:10.974 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-10-15T16:39:10.974 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "test-zone1.rgw.control", 2020-10-15T16:39:10.974 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-10-15T16:39:10.974 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-10-15T16:39:10.974 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "test-zone1.rgw.log", 2020-10-15T16:39:10.974 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-10-15T16:39:10.975 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:10.976 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:10.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:10.978 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-10-15T16:39:10.979 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:10.980 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-10-15T16:39:11.937 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-10-15T16:39:11.955 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-10-15T16:39:12.955 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-10-15T16:39:12.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:12.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:12.980 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:13.021 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:13.021 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:13.043 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:13.043 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:13.043 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:39:13.043 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "test-zone1.rgw.control", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "test-zone1.rgw.log", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-10-15T16:39:13.044 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-10-15T16:39:13.045 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:39:13.046 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: "compression_type": "random" 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:13.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "", 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-10-15T16:39:13.048 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:13.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:13.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:13.050 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:13.127 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:13.127 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:21.201 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:21.202 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:21.202 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 1, 2020-10-15T16:39:21.202 INFO:teuthology.orchestra.run.smithi183.stdout: "predecessor_uuid": "dfa1fd57-6cc1-40c3-8be1-b0a367d9bbab", 2020-10-15T16:39:21.202 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": [], 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: "period_map": { 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: "zonegroups": [ 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:39:21.203 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:21.204 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:21.205 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "false", 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:39:21.206 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:39:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: "short_zone_ids": [ 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:21.209 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 770630339 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: "period_config": { 2020-10-15T16:39:21.210 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:21.211 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:21.212 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:21.213 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_name": "test-realm", 2020-10-15T16:39:21.213 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:39:21.213 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:21.214 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-10-15T16:39:21.214 INFO:teuthology.orchestra.run:waiting for 300 2020-10-15T16:39:21.255 INFO:tasks.rgw.c1.client.0.smithi183.stdout:2020-10-15T16:39:21.253+0000 7f65e2ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 15485) UID: 0 2020-10-15T16:39:21.256 INFO:tasks.rgw.c1.client.0.smithi183.stdout:2020-10-15T16:39:21.253+0000 7f6606a50100 -1 shutting down 2020-10-15T16:39:21.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-10-15T16:39:27.316 INFO:tasks.rgw.c1.client.0:Stopped 2020-10-15T16:39:27.317 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-10-15T16:39:27.317 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-10-15T16:39:27.320 INFO:tasks.rgw.c1.client.0:Started 2020-10-15T16:39:27.320 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8000 2020-10-15T16:39:27.336 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:27.337 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:27.341 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi183.front.sepia.ceph.com port 8000: Connection refused 2020-10-15T16:39:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-15T16:39:27.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2020-10-15T16:39:28.343 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8000 2020-10-15T16:39:28.358 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:28.359 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:28.365 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2020-10-15T16:39:28.366 INFO:teuthology.orchestra.run.smithi183.stdout:anonymous 2020-10-15T16:39:28.366 INFO:tasks.rgw_multisite:creating admin user.. 2020-10-15T16:39:28.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:28.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:28.367 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:28.452 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:28.452 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:28.687 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:28.688 INFO:teuthology.orchestra.run.smithi183.stdout: "user_id": "realm-admin", 2020-10-15T16:39:28.688 INFO:teuthology.orchestra.run.smithi183.stdout: "display_name": "Realm Admin", 2020-10-15T16:39:28.688 INFO:teuthology.orchestra.run.smithi183.stdout: "email": "", 2020-10-15T16:39:28.688 INFO:teuthology.orchestra.run.smithi183.stdout: "suspended": 0, 2020-10-15T16:39:28.688 INFO:teuthology.orchestra.run.smithi183.stdout: "max_buckets": 1000, 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: "subusers": [], 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: "keys": [ 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: "user": "realm-admin", 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:28.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: "swift_keys": [], 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: "caps": [], 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: "op_mask": "read, write, delete", 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: "system": "true", 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "", 2020-10-15T16:39:28.690 INFO:teuthology.orchestra.run.smithi183.stdout: "default_storage_class": "", 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_tags": [], 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:39:28.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:39:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:28.693 INFO:teuthology.orchestra.run.smithi183.stdout: "temp_url_keys": [], 2020-10-15T16:39:28.693 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "rgw", 2020-10-15T16:39:28.693 INFO:teuthology.orchestra.run.smithi183.stdout: "mfa_ids": [] 2020-10-15T16:39:28.693 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:28.693 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:39:28.719 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-10-15T16:39:28.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi183.front.sepia.ceph.com:8000', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:28.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi183.front.sepia.ceph.com:8000', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:28.721 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi183.front.sepia.ceph.com:8000 --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:28.904 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:39:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-realm", 2020-10-15T16:39:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "current_period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 2 2020-10-15T16:39:28.905 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:39:28.906 INFO:tasks.rgw_multisite:creating zone.. 2020-10-15T16:39:28.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi038.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:28.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi038.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:28.907 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --endpoints http://smithi038.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:28.994 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:28.994 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:29.061 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:29.057+0000 7f0a366360c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:39:29.062 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:39:29.057+0000 7f0a366360c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:39:29.080 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:39:29.080 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:39:29.080 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:39:29.080 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:39:29.080 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:39:29.081 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:39:29.082 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:39:29.083 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:39:29.084 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:39:29.084 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:39:29.084 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-10-15T16:39:29.084 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:29.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:29.084 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:39:29.085 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:39:29.086 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-10-15T16:39:30.149 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-10-15T16:39:30.166 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-10-15T16:39:31.152 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-10-15T16:39:31.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:31.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:31.183 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:31.244 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:39:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:39:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:39:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:39:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:39:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:39:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:39:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:39:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-15T16:39:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "", 2020-10-15T16:39:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:39:31.250 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:39:31.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:31.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:31.251 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:31.331 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:31.332 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:33.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-10-15T16:39:39.361 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:39:39.464 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:39:39.464 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:39.464 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 2, 2020-10-15T16:39:39.464 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "dfa1fd57-6cc1-40c3-8be1-b0a367d9bbab", 2020-10-15T16:39:39.464 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [], 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:39:39.465 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:39:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:39:39.467 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:39:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:39:39.469 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:39:39.470 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:39:39.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:39:39.472 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:39:39.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:39:39.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:39:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:39:39.476 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:39:39.477 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:39:39.478 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:39:39.479 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-10-15T16:39:39.479 INFO:teuthology.orchestra.run:waiting for 300 2020-10-15T16:39:39.523 INFO:tasks.rgw.c2.client.0.smithi038.stdout:2020-10-15T16:39:39.521+0000 7fd390c3e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 15479) UID: 0 2020-10-15T16:39:39.523 INFO:tasks.rgw.c2.client.0.smithi038.stdout:2020-10-15T16:39:39.521+0000 7fd3a7bec100 -1 shutting down 2020-10-15T16:39:42.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-10-15T16:39:45.580 INFO:tasks.rgw.c2.client.0:Stopped 2020-10-15T16:39:45.581 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-10-15T16:39:45.581 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-10-15T16:39:45.584 INFO:tasks.rgw.c2.client.0:Started 2020-10-15T16:39:45.584 INFO:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:8000 2020-10-15T16:39:45.600 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:45.601 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:45.605 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi038.front.sepia.ceph.com port 8000: Connection refused 2020-10-15T16:39:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-15T16:39:46.607 INFO:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:8000 2020-10-15T16:39:46.624 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:39:46.625 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:39:46.631 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-10-15T16:39:46.632 INFO:teuthology.orchestra.run.smithi038.stdout:anonymous 2020-10-15T16:39:46.633 INFO:tasks.rgw_multisite:creating zone.. 2020-10-15T16:39:46.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi183.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:46.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi183.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:46.634 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --endpoints http://smithi183.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:46.680 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:46.680 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:46.738 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:39:46.733+0000 7f1e042c80c0 0 failed reading obj info from .rgw.root:zone_info.669d2e2a-b26c-4898-99f7-fa22d626fe93: (2) No such file or directory 2020-10-15T16:39:46.738 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:39:46.733+0000 7f1e042c80c0 0 WARNING: could not read zone params for zone id=669d2e2a-b26c-4898-99f7-fa22d626fe93 name=test-zone2 2020-10-15T16:39:46.748 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:46.748 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:39:46.749 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:39:46.749 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-10-15T16:39:46.749 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "test-zone3.rgw.control", 2020-10-15T16:39:46.749 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-10-15T16:39:46.749 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-10-15T16:39:46.749 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "test-zone3.rgw.log", 2020-10-15T16:39:46.750 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-10-15T16:39:46.750 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-10-15T16:39:46.750 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-10-15T16:39:46.750 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-10-15T16:39:46.751 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-10-15T16:39:46.751 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-10-15T16:39:46.752 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-10-15T16:39:46.752 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-10-15T16:39:46.752 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-10-15T16:39:46.752 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:39:46.752 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:39:46.753 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-10-15T16:39:46.754 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:39:46.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:46.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:46.755 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:46.755 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:46.755 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-10-15T16:39:46.755 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:46.757 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-10-15T16:39:47.968 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-10-15T16:39:47.994 INFO:teuthology.orchestra.run.smithi183:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-10-15T16:39:48.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-10-15T16:39:48.990 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-10-15T16:39:49.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:49.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:49.010 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:49.054 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:49.055 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:49.077 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:39:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:39:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-10-15T16:39:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "test-zone3.rgw.control", 2020-10-15T16:39:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-10-15T16:39:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-10-15T16:39:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "test-zone3.rgw.log", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-10-15T16:39:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:49.080 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:39:49.081 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-10-15T16:39:49.082 INFO:teuthology.orchestra.run.smithi183.stdout: "compression_type": "random" 2020-10-15T16:39:49.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:49.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:49.082 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-10-15T16:39:49.082 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:39:49.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:49.083 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:49.083 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:49.083 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "", 2020-10-15T16:39:49.083 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-10-15T16:39:49.083 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:49.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:49.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:39:49.085 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:39:49.119 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:39:49.120 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:39:57.191 INFO:teuthology.orchestra.run.smithi183.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:39:57.252 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:39:57.252 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:57.252 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 3, 2020-10-15T16:39:57.252 INFO:teuthology.orchestra.run.smithi183.stdout: "predecessor_uuid": "dfa1fd57-6cc1-40c3-8be1-b0a367d9bbab", 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": [], 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: "period_map": { 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: "zonegroups": [ 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:39:57.253 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:39:57.254 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:39:57.254 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:39:57.254 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:57.254 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:57.254 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.254 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone2", 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:57.255 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:39:57.256 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.256 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:39:57.256 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:39:57.256 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:39:57.256 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:39:57.256 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:39:57.257 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:39:57.257 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:39:57.257 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:39:57.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.257 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:39:57.258 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:39:57.260 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:39:57.261 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:39:57.262 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:39:57.263 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: "short_zone_ids": [ 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 757601199 2020-10-15T16:39:57.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 1274429389 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 770630339 2020-10-15T16:39:57.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: "period_config": { 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:39:57.266 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:39:57.267 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:39:57.278 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:39:57.279 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:39:57.279 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:39:57.279 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:39:57.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:39:57.279 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:39:57.279 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_name": "test-realm", 2020-10-15T16:39:57.280 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:39:57.280 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:39:57.281 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-10-15T16:39:57.281 INFO:teuthology.orchestra.run:waiting for 300 2020-10-15T16:39:57.322 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:39:57.321+0000 7fa386e35700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 15487) UID: 0 2020-10-15T16:39:57.323 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:39:57.321+0000 7fa39e5e4100 -1 shutting down 2020-10-15T16:40:01.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-10-15T16:40:03.383 INFO:tasks.rgw.c1.client.1:Stopped 2020-10-15T16:40:03.384 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-10-15T16:40:03.384 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-10-15T16:40:03.387 INFO:tasks.rgw.c1.client.1:Started 2020-10-15T16:40:03.387 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8001 2020-10-15T16:40:03.404 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:40:03.404 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:40:03.408 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi183.front.sepia.ceph.com port 8001: Connection refused 2020-10-15T16:40:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-15T16:40:04.410 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8001 2020-10-15T16:40:04.424 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:40:04.424 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:40:04.426 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2020-10-15T16:40:04.427 INFO:teuthology.orchestra.run.smithi183.stdout:anonymous 2020-10-15T16:40:04.427 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-10-15T16:40:04.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi183.front.sepia.ceph.com:8000', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--url', 'http://smithi183.front.sepia.ceph.com:8000', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:04.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi183.front.sepia.ceph.com:8000', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--url', 'http://smithi183.front.sepia.ceph.com:8000', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:04.428 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi183.front.sepia.ceph.com:8000 --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --url http://smithi183.front.sepia.ceph.com:8000 --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:40:04.477 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:04.478 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:04.535 INFO:tasks.rgw.c2.client.0.smithi038.stdout:2020-10-15T16:40:04.533+0000 7feeede0b700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.9e20c2cc-422b-4644-9bb7-b97bb5e18239.control err=-107 2020-10-15T16:40:04.575 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:04.576 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:40:04.576 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-realm", 2020-10-15T16:40:04.576 INFO:teuthology.orchestra.run.smithi038.stdout: "current_period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:04.576 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 2 2020-10-15T16:40:04.576 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:04.578 INFO:tasks.rgw_multisite:creating zone.. 2020-10-15T16:40:04.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi038.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:04.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--endpoints', 'http://smithi038.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:04.579 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --endpoints http://smithi038.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:40:04.631 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:04.632 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:04.701 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:40:04.697+0000 7f87cfd6b0c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:40:04.702 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:40:04.697+0000 7f87cfd6b0c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:40:04.703 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:40:04.701+0000 7f87cfd6b0c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:40:04.703 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:40:04.701+0000 7f87cfd6b0c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:40:04.715 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:04.715 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:40:04.715 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:40:04.716 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-10-15T16:40:04.716 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone4.rgw.control", 2020-10-15T16:40:04.716 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-10-15T16:40:04.717 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-10-15T16:40:04.717 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone4.rgw.log", 2020-10-15T16:40:04.717 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-10-15T16:40:04.717 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-10-15T16:40:04.718 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-10-15T16:40:04.718 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-10-15T16:40:04.718 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-10-15T16:40:04.719 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-10-15T16:40:04.719 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-10-15T16:40:04.719 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-10-15T16:40:04.719 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-10-15T16:40:04.720 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:40:04.720 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:40:04.720 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:40:04.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:04.721 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:40:04.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:04.721 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:40:04.722 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:04.722 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-10-15T16:40:04.722 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:40:04.722 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:40:04.723 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-10-15T16:40:04.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:04.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-10-15T16:40:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:40:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_config": { 2020-10-15T16:40:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event_retention_days": 7, 2020-10-15T16:40:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "start_with_full_sync": false 2020-10-15T16:40:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:40:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-10-15T16:40:04.727 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:04.729 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-10-15T16:40:05.150 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-10-15T16:40:05.163 INFO:teuthology.orchestra.run.smithi038:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-10-15T16:40:06.141 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-10-15T16:40:06.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:06.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:06.167 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:40:06.231 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:06.268 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:06.269 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:06.269 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone4.rgw.control", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone4.rgw.log", 2020-10-15T16:40:06.270 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-10-15T16:40:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:40:06.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-10-15T16:40:06.273 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:06.274 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:06.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_config": { 2020-10-15T16:40:06.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event_retention_days": 7, 2020-10-15T16:40:06.275 INFO:teuthology.orchestra.run.smithi038.stdout: "start_with_full_sync": false 2020-10-15T16:40:06.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:06.275 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "", 2020-10-15T16:40:06.276 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-10-15T16:40:06.276 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:06.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:06.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:06.277 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:40:06.314 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:06.314 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:07.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-10-15T16:40:14.363 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:40:14.436 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:14.437 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:14.437 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 4, 2020-10-15T16:40:14.437 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "dfa1fd57-6cc1-40c3-8be1-b0a367d9bbab", 2020-10-15T16:40:14.437 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [], 2020-10-15T16:40:14.437 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:40:14.438 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:14.438 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:40:14.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.438 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:40:14.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:40:14.438 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:40:14.439 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:40:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:40:14.441 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:40:14.442 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:40:14.443 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:40:14.444 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:40:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:40:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:40:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:40:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:40:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:40:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:40:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:40:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:40:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:40:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:40:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.610 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:40:14.611 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:40:14.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.611 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:40:14.611 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:14.612 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:40:14.613 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:40:14.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:14.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:14.616 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:40:14.616 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:40:14.616 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:14.616 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:14.617 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-10-15T16:40:14.617 INFO:teuthology.orchestra.run:waiting for 300 2020-10-15T16:40:14.658 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T16:40:14.657+0000 7f38a6ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 15482) UID: 0 2020-10-15T16:40:14.658 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T16:40:14.657+0000 7f38caf22100 -1 shutting down 2020-10-15T16:40:16.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-10-15T16:40:20.718 INFO:tasks.rgw.c2.client.1:Stopped 2020-10-15T16:40:20.719 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-10-15T16:40:20.719 INFO:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-10-15T16:40:20.722 INFO:tasks.rgw.c2.client.1:Started 2020-10-15T16:40:20.722 INFO:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:8001 2020-10-15T16:40:20.740 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:40:20.740 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:40:20.744 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi038.front.sepia.ceph.com port 8001: Connection refused 2020-10-15T16:40:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-15T16:40:21.746 INFO:teuthology.orchestra.run.smithi038:> curl http://smithi038.front.sepia.ceph.com:8001 2020-10-15T16:40:21.763 INFO:teuthology.orchestra.run.smithi038.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:40:21.763 INFO:teuthology.orchestra.run.smithi038.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:40:21.768 INFO:teuthology.orchestra.run.smithi038.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-10-15T16:40:21.768 INFO:teuthology.orchestra.run.smithi038.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005f887b75-10ea-test-zone4","HostId":"10ea-test-zone4-test-zonegroup"} 2020-10-15T16:40:21.769 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-10-15T16:40:21.769 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-10-15T16:40:21.837 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-10-15T16:40:21.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:21.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:40:21.838 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:40:21.884 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:21.885 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:22.093 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:40:22.094 INFO:teuthology.orchestra.run.smithi183.stdout: "user_id": "rgw-multisite-test-user", 2020-10-15T16:40:22.094 INFO:teuthology.orchestra.run.smithi183.stdout: "display_name": "Multisite Test User", 2020-10-15T16:40:22.094 INFO:teuthology.orchestra.run.smithi183.stdout: "email": "", 2020-10-15T16:40:22.094 INFO:teuthology.orchestra.run.smithi183.stdout: "suspended": 0, 2020-10-15T16:40:22.094 INFO:teuthology.orchestra.run.smithi183.stdout: "max_buckets": 1000, 2020-10-15T16:40:22.094 INFO:teuthology.orchestra.run.smithi183.stdout: "subusers": [], 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: "keys": [ 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: "user": "rgw-multisite-test-user", 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "6406IOSIAABGNMPGQTIO", 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "kEnQSiIXWQRWjVEMbT5UK3LdxVSw76ITKjorkFQi" 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:40:22.095 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "swift_keys": [], 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "caps": [], 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "op_mask": "read, write, delete", 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "", 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "default_storage_class": "", 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_tags": [], 2020-10-15T16:40:22.096 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:40:22.097 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:40:22.098 INFO:teuthology.orchestra.run.smithi183.stdout: "temp_url_keys": [], 2020-10-15T16:40:22.099 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "rgw", 2020-10-15T16:40:22.099 INFO:teuthology.orchestra.run.smithi183.stdout: "mfa_ids": [] 2020-10-15T16:40:22.099 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:40:22.099 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T16:40:22.121 INFO:rgw_multi.tests:meta checkpoint 2020-10-15T16:40:22.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:22.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:22.123 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:22.203 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:22.203 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:22.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-10-15T16:40:22.430 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:40:22.462 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:40:22.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:40:22.462 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:40:22.087940Z" 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:40:22.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:40:22.464 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:40:22.465 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:40:22.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:40:22.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:22.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:22.467 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:22.507 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:22.507 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:22.775 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:22.777 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:22.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:22.779 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:22.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:22.781 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:22.782 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:22.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:22.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:22.785 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:22.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:27.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:27.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:27.788 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:27.829 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:27.829 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:28.096 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:28.096 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:28.096 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:28.096 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:28.097 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:28.097 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:28.097 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:28.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:28.097 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:28.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:28.098 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:28.098 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:28.098 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:28.098 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:28.098 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:28.098 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:28.099 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:28.100 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:28.101 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:28.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:28.103 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:28.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:28.105 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:28.105 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:28.105 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:28.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:28.105 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:28.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:33.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:33.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:33.109 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:33.148 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:33.149 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:33.419 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:33.419 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:33.422 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:33.422 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:33.422 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:33.422 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:33.423 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:33.423 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:33.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:33.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:33.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:33.423 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:33.424 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:33.425 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:33.426 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:33.426 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:33.426 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:33.426 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:33.426 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:33.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:33.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:33.429 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:33.429 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:33.429 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:33.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:33.429 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:33.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:38.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:38.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:38.432 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:38.730 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:38.739 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:38.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:43.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:43.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:43.742 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:43.785 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:43.785 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:44.041 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:44.042 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:44.042 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:44.042 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:44.042 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:44.042 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:44.043 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:44.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:44.045 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:44.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:44.047 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:44.048 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:44.049 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:44.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:44.051 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:44.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:49.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:49.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:49.053 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:49.096 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:49.096 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:49.374 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:49.383 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:49.384 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:54.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:54.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:54.386 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:54.428 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:54.429 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:40:54.707 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:40:54.707 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:40:54.707 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:40:54.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:40:54.709 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:54.710 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:54.710 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:54.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:54.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:54.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:54.710 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:40:54.711 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:54.711 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:54.711 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:54.711 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:54.711 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:54.711 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:40:54.712 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:54.713 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:54.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:54.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:54.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:40:54.714 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:40:54.714 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:40:54.714 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:40:54.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:40:54.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:40:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:40:54.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:40:59.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:59.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:40:59.720 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:40:59.759 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:40:59.759 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:00.044 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:00.054 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:00.055 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:00.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:00.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:05.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:05.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:05.059 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:05.099 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:05.099 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:05.353 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:05.363 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:05.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:10.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:10.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:10.366 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:10.407 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:10.408 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:10.674 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:10.674 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:10.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:10.676 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:10.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:10.679 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:10.680 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:10.681 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:10.682 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:10.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:10.683 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:10.683 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:10.683 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:10.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:10.683 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:10.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:15.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:15.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:15.686 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:15.726 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:15.726 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:15.975 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:15.981 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:15.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:15.984 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:15.985 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:15.985 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:15.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:15.985 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:15.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:20.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:20.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:20.988 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:21.031 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:21.031 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:21.291 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:21.291 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:21.292 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:21.292 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:21.292 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:21.292 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:21.292 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:21.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:21.293 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:21.296 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:21.296 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:21.296 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:21.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:21.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:21.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:21.297 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:21.299 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:21.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:21.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:21.300 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:21.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:21.300 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:21.301 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:21.301 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:21.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:21.301 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:21.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:26.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:26.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:26.305 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:26.349 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:26.349 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:26.606 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:26.607 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:26.607 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:26.607 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:26.607 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:26.607 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:26.608 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:26.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:26.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:26.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:26.610 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:26.610 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:26.610 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:26.610 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:26.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:26.612 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:26.613 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:26.614 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:26.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:26.616 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:26.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:31.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:31.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:31.619 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:31.662 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:31.663 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:31.916 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:31.916 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:31.916 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:31.916 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:31.917 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:31.917 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:31.917 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:31.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:31.917 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:31.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:31.918 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:31.918 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:31.918 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:31.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:31.920 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:31.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:31.922 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:31.922 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:31.922 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:31.922 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:31.922 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:31.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:31.926 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:31.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:31.926 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:31.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:36.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:36.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:36.930 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:37.247 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:37.257 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:37.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:42.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:42.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:42.260 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:42.301 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:42.301 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:42.561 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:42.561 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:42.561 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:42.561 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:42.562 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:42.563 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:42.564 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:42.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:42.566 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:42.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:42.568 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:42.569 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:42.570 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:42.570 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:42.570 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:42.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:47.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:47.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:47.573 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:47.615 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:47.616 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:47.892 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:47.892 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:47.892 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:47.892 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:47.893 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:47.893 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:47.893 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:47.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:47.893 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:47.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:47.894 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:47.895 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:47.896 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:47.897 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:47.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:47.899 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:47.900 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:47.901 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:47.901 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:47.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:47.901 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:47.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:52.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:52.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:52.904 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:52.944 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:52.944 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:53.204 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:53.205 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:53.205 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:53.205 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:53.205 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:53.206 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:53.206 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:53.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:53.207 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:53.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:53.207 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:53.207 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:53.208 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:53.208 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:53.208 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:53.208 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:53.209 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:53.209 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:53.209 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:53.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:53.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:53.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:53.210 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:53.211 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:53.211 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:53.211 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:53.211 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:53.212 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:53.212 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:53.212 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:53.213 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:53.213 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:53.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:53.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:53.214 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:53.214 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:53.214 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:53.215 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:53.215 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:53.215 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:53.215 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:53.216 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:53.216 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:53.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:53.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:53.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:53.217 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:53.217 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:53.218 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:53.218 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:53.218 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:53.218 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:53.219 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:53.219 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:53.219 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:53.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:53.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:53.220 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:53.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:53.220 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:53.220 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:53.221 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:53.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:53.221 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:53.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:41:58.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:58.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:41:58.225 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:41:58.265 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:41:58.265 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:41:58.548 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:41:58.549 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:41:58.549 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:41:58.549 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:41:58.549 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:41:58.549 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:41:58.550 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:58.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:58.550 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:41:58.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:58.550 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:41:58.550 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:58.552 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:41:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:41:58.555 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:41:58.556 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:41:58.556 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:41:58.556 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:41:58.556 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:41:58.556 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:41:58.556 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:41:58.557 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:41:58.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:41:58.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:58.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:58.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:41:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:41:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:41:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:41:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:41:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:41:58.559 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:41:58.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:42:03.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:03.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:03.562 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:03.605 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:03.605 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:03.867 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:03.868 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:03.868 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:03.868 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:03.869 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:03.869 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:03.869 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:03.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:03.870 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:03.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:03.871 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:03.871 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:03.871 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:03.871 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:03.872 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:03.872 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:03.872 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:03.873 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:03.873 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:03.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:03.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:03.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:03.874 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:03.874 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:03.875 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:03.875 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:03.875 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:03.876 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:03.876 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:03.876 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:03.876 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:03.877 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:03.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:03.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:03.878 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:03.878 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:03.878 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:03.878 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:03.879 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:03.879 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:03.879 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:03.886 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:03.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:42:08.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:08.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:08.889 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:08.930 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:08.930 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:09.188 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:09.198 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:09.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:42:14.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:14.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:14.201 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:14.237 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:14.238 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:14.508 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:14.509 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:14.509 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:14.509 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:14.509 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:14.510 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:14.510 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:14.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:14.510 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:14.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:14.511 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:14.512 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:14.512 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:14.512 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:14.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:14.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:14.512 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:14.513 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:14.514 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:14.515 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:14.515 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:14.515 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:14.515 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:14.515 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:14.515 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:14.516 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:14.517 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:14.517 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:14.517 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:14.517 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:14.517 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:14.517 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:14.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:14.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:14.518 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:14.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:14.518 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:14.518 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:14.519 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:14.519 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:14.519 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:14.520 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602780022.087940_55.1 target= 2020-10-15T16:42:19.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:19.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:19.523 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:19.565 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:19.566 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:19.814 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:19.814 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:19.814 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:19.815 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:19.815 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:19.815 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:19.815 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:19.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:19.815 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:19.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:19.816 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:19.816 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:19.816 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:19.816 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:19.816 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.087940Z", 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:19.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:19.818 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:19.818 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:19.818 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:19.818 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:19.818 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:19.818 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:19.819 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:19.820 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:19.821 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:19.821 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:19.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:19.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:19.821 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:19.821 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:19.822 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:19.823 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:19.824 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:19.824 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:19.825 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:19.825 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:19.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:19.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:19.826 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:19.867 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:19.867 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:20.115 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:20.115 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:20.116 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:20.116 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:20.116 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:20.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.118 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:20.118 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:20.119 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:20.119 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:20.119 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:20.120 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:20.120 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:20.120 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:40:22.087940Z", 2020-10-15T16:42:20.120 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:20.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.122 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:20.122 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:20.122 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:20.122 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:20.123 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:20.123 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:20.123 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:20.124 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:20.124 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:20.124 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.125 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:20.125 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:20.126 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:20.126 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:20.126 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:20.127 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:20.127 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:20.127 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:20.127 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:20.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.129 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:20.129 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:20.129 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:20.129 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:20.130 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:20.130 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:20.130 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:20.131 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:20.131 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:20.131 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.131 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.132 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:20.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.132 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:20.133 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:20.133 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:20.133 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.133 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:20.135 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:20.136 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:20.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:20.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:20.137 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:20.444 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:20.445 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:20.445 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:20.445 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:20.446 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:20.446 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:20.446 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:20.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:20.447 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:20.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:20.451 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:20.452 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:20.452 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:20.452 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:20.452 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:20.452 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:20.453 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:20.453 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.087940Z", 2020-10-15T16:42:20.453 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:20.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:20.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:20.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:20.454 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:20.454 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:20.455 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:20.456 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:20.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:20.458 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:20.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:20.460 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:20.460 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:20.460 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:20.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:20.460 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:20.461 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:20.462 INFO:rgw_multi.tests:meta checkpoint 2020-10-15T16:42:20.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:20.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:20.463 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:20.502 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:20.502 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:20.733 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:20.764 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:40:22.087940Z" 2020-10-15T16:42:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:20.766 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:20.767 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:20.767 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:20.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:20.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:20.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:20.769 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:20.810 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:20.811 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:21.070 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:21.071 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:21.071 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:21.071 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:21.071 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:21.071 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.072 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.087940Z", 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:21.074 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:21.076 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:21.077 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.078 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.080 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:21.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:21.081 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:21.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:21.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:21.082 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:21.122 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:21.123 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:21.363 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:21.364 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:21.364 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:21.364 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:21.364 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:21.364 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:21.364 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:21.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:21.365 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:21.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:21.365 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:21.365 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:21.365 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:40:22.087940Z", 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:21.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:21.367 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:21.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:21.370 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:21.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:21.372 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:21.373 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:21.373 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:21.374 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:21.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:21.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:21.375 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:21.416 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:21.417 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:21.688 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:21.688 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.689 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:21.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.690 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:21.690 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.690 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.690 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780022.087940_55.1", 2020-10-15T16:42:21.690 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.087940Z", 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:21.692 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:21.693 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.694 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:21.695 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:21.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:21.697 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:21.697 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:21.697 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:21.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:21.697 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:21.698 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:21.699 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-10-15T16:42:21.707 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-1 2020-10-15T16:42:25.040 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-2 2020-10-15T16:42:28.204 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-3 2020-10-15T16:42:28.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:28.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:28.287 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:28.319 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:28.319 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:28.520 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:28.554 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:28.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:28.555 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780148.269888_71.1", 2020-10-15T16:42:28.555 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:28.269888Z" 2020-10-15T16:42:28.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:28.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:28.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:28.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:28.557 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:28.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:28.557 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:28.557 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:28.558 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:28.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:28.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:28.560 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:28.601 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:28.861 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:28.861 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:28.861 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:28.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780148.269888_71.1", 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:28.863 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.269888Z", 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:28.864 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:28.865 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:28.866 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:28.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:28.868 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:28.869 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:28.870 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:28.870 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:28.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:28.870 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:28.871 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:28.871 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:28.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:28.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:28.873 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:28.911 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:28.912 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:29.174 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:29.174 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:29.174 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:29.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780148.269888_71.1", 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:29.176 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.269888Z", 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:29.177 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:29.178 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:29.179 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:29.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:29.181 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:29.182 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:29.183 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:29.183 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:29.183 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:29.183 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:29.184 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:29.185 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:29.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:29.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:29.186 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:29.222 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:29.223 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:29.491 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:29.491 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:29.492 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780148.269888_71.1", 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:29.493 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.269888Z", 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:29.494 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:29.495 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:29.496 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:29.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:29.498 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:29.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:29.500 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:29.500 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:29.500 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:29.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:29.500 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:29.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:29.543 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-10-15T16:42:29.545 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-4 2020-10-15T16:42:29.578 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-5 2020-10-15T16:42:29.650 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-6 2020-10-15T16:42:29.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:29.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:29.719 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:29.760 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:29.761 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:29.975 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:30.008 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:30.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.008 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:30.009 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:29.701238Z" 2020-10-15T16:42:30.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.009 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:30.009 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:30.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.010 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:30.010 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:30.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.011 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:30.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:30.011 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.011 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:30.012 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:30.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:30.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:30.013 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:30.054 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:30.055 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:30.287 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:30.287 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:30.287 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:30.287 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:30.287 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.701238Z", 2020-10-15T16:42:30.289 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.290 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:30.290 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.290 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.292 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.293 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.294 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:30.295 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:30.296 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:30.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.296 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:30.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:30.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:30.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:30.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:30.298 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:30.340 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:30.340 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:30.576 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:30.576 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:30.577 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:30.577 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:30.577 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:30.577 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:30.577 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:30.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:30.578 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.701238Z", 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:30.580 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:30.581 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:30.582 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:30.582 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:30.582 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:30.582 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:30.582 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.582 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:30.583 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:30.583 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:30.583 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:30.583 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:30.584 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:30.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:30.586 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:30.587 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:30.587 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:30.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:30.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:30.588 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:30.898 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.701238Z", 2020-10-15T16:42:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:30.908 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:30.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:31.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:31.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:31.246 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:31.279 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:31.279 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:31.486 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:31.516 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:31.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:31.516 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:31.516 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:29.701238Z" 2020-10-15T16:42:31.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:31.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:31.517 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:31.517 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:31.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:31.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:31.517 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:31.518 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:31.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:31.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:31.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:31.521 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:31.563 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:31.563 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:31.828 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:31.829 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:31.829 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:31.829 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:31.829 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:31.829 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:31.830 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.701238Z", 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:31.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:31.832 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:31.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:31.834 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:31.835 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:31.835 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:31.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:31.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:31.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:31.835 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:31.836 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:31.837 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:31.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:31.838 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:31.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:31.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:31.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:31.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:31.841 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:31.881 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:31.881 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:32.132 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:32.132 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:32.132 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:32.132 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:32.132 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:32.133 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.701238Z", 2020-10-15T16:42:32.134 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:32.135 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:32.137 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.138 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:32.139 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:32.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.140 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:32.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.140 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:32.140 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:32.141 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:32.141 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.141 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:32.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:32.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:32.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:32.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:32.144 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:32.178 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:32.178 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:32.537 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:32.538 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:32.538 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:32.538 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:32.538 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:32.538 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:32.538 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:32.539 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780149.701238_85.1", 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.701238Z", 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:32.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:32.541 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:32.542 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:32.543 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:32.544 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:32.547 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:32.547 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:32.547 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:32.547 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:32.548 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:32.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:32.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:32.548 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:32.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:32.549 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:32.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:32.549 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:32.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:32.549 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:32.550 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:32.592 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-10-15T16:42:32.593 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-7 2020-10-15T16:42:32.623 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-8 2020-10-15T16:42:32.678 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-9 2020-10-15T16:42:32.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:32.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:32.732 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:32.765 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:32.766 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:32.961 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:32.988 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:32.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.989 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780152.720290_101.1", 2020-10-15T16:42:32.989 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:32.720290Z" 2020-10-15T16:42:32.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.989 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:32.989 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:32.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.990 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:32.990 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:32.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:32.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:32.991 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:32.991 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:32.991 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:32.992 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:32.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:32.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:32.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:32.994 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:33.033 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:33.034 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:33.276 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:33.276 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:33.276 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:33.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780152.720290_101.1", 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:33.278 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.720290Z", 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:33.279 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.280 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:33.281 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:33.283 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:33.285 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:33.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.285 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:33.286 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:33.286 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:33.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:33.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:33.288 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:33.328 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:33.329 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:33.573 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:33.573 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:33.574 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:33.574 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:33.574 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:33.574 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:33.574 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:33.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780152.720290_101.1", 2020-10-15T16:42:33.575 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:33.576 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:33.576 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:33.576 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.720290Z", 2020-10-15T16:42:33.576 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:33.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:33.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:33.577 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:33.578 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:33.579 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:33.579 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:33.579 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:33.579 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:33.579 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:33.579 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:33.580 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:33.581 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:33.582 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:33.583 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:33.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:33.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:33.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:33.585 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:33.889 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:33.890 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:33.890 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:33.890 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:33.890 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:33.890 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780152.720290_101.1", 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.891 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:33.892 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.892 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.720290Z", 2020-10-15T16:42:33.892 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.893 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:33.896 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:33.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:33.898 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:33.898 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:33.898 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:33.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:33.898 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:33.899 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:34.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:34.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:34.194 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:34.228 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:34.228 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:34.432 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:34.463 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:42:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:34.465 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:34.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:34.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:34.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:34.467 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:34.508 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:34.508 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:34.780 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:34.789 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:34.790 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:34.791 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:34.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:34.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:34.792 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:34.836 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:34.837 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:35.069 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:35.069 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:35.069 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:35.069 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:35.070 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:35.070 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:35.070 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:35.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:35.070 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:35.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:35.071 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:35.072 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:35.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:35.074 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:35.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:35.076 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:35.077 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:35.078 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:35.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:35.078 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:35.079 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:35.079 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:35.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:35.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:35.080 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:35.121 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:35.121 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:35.388 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:35.397 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:35.398 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:35.413 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-10-15T16:42:35.413 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-10 2020-10-15T16:42:35.447 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-11 2020-10-15T16:42:35.514 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-12 2020-10-15T16:42:36.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:36.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:36.160 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:36.204 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:36.204 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:36.435 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:36.468 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:36.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:36.469 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:36.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:42:36.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:36.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:36.469 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780155.557492_88.1", 2020-10-15T16:42:36.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:35.557492Z" 2020-10-15T16:42:36.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:36.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:36.470 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:36.470 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:36.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:36.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:36.471 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:36.471 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:36.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:36.471 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:36.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:36.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:36.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:36.474 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:36.517 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:36.517 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:36.719 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780155.557492_88.1", 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.557492Z", 2020-10-15T16:42:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:36.728 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:36.729 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:36.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:36.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:36.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:36.731 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:36.775 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:36.775 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:37.026 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780155.557492_88.1", 2020-10-15T16:42:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.557492Z", 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:37.031 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:37.032 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:37.033 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:37.034 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:37.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:37.035 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:37.035 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:37.035 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:37.035 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:37.035 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:37.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:37.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:37.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:37.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:37.038 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:37.079 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:37.079 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:37.345 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780155.557492_88.1", 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.557492Z", 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:37.354 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:37.355 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:37.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:37.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:37.357 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:37.446 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:37.697 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:37.697+0000 7fd2d3eb50c0 1 bucket sync caught up with source: 2020-10-15T16:42:37.697 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.5.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.5.5, , ] 2020-10-15T16:42:37.697 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000001.5.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.5.5, , ] 2020-10-15T16:42:37.697 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:37.697+0000 7fd2d3eb50c0 0 bucket checkpoint complete 2020-10-15T16:42:37.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-10 zones={test-zone2, test-zone1} 2020-10-15T16:42:37.738 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:37.746 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:37.746 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:37.747 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:37.747 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:37.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:37.753 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:37.754 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:37.754 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:37.754 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:37.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:37.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:37.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:38.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:38.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:38.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:38.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:38.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:38.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:38.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:38.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:38.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:38.854 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-10 zones={test-zone2, test-zone1} 2020-10-15T16:42:38.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:38.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:38.855 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:38.899 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:38.899 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:39.124 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:39.125+0000 7fca4e0ca0c0 1 bucket sync caught up with source: 2020-10-15T16:42:39.125 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.5.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.5.5, , ] 2020-10-15T16:42:39.125 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.5.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.5.5, , ] 2020-10-15T16:42:39.125 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:39.125+0000 7fca4e0ca0c0 0 bucket checkpoint complete 2020-10-15T16:42:39.149 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-10 zones={test-zone3, test-zone1} 2020-10-15T16:42:39.158 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:39.206 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:39.207 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:39.207 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:39.208 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:39.208 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:39.215 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:39.215 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:39.216 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:39.216 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:39.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:39.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:39.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:39.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:39.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:39.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:39.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:39.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:39.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:40.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:40.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:40.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:40.413 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-10 zones={test-zone3, test-zone1} 2020-10-15T16:42:40.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:40.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:40.414 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:40.464 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:40.464 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:40.731 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:40.733+0000 7fdfc4cd20c0 1 bucket sync caught up with source: 2020-10-15T16:42:40.732 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.5.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.5.5, , ] 2020-10-15T16:42:40.732 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000001.5.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.5.5, , ] 2020-10-15T16:42:40.732 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:40.733+0000 7fdfc4cd20c0 0 bucket checkpoint complete 2020-10-15T16:42:40.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:40.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:40.837 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:40.885 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:40.885 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:41.099 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:41.101+0000 7fd3dbf340c0 1 bucket sync caught up with source: 2020-10-15T16:42:41.099 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.13.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2020-10-15T16:42:41.099 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.13.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2020-10-15T16:42:41.099 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:41.101+0000 7fd3dbf340c0 0 bucket checkpoint complete 2020-10-15T16:42:41.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-11 zones={test-zone1, test-zone2} 2020-10-15T16:42:41.139 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:41.146 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:41.146 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:41.147 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:41.147 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:41.147 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:41.154 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:41.154 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:41.155 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:41.155 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:41.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:41.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:41.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:41.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:41.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:41.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:41.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:41.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:41.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:41.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:42.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:42.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:42.260 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-11 zones={test-zone1, test-zone2} 2020-10-15T16:42:42.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:42.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:42.262 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:42.308 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:42.309 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:42.544 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:42.545+0000 7ff4985f60c0 1 bucket sync caught up with source: 2020-10-15T16:42:42.545 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.13.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2020-10-15T16:42:42.545 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.13.5, , 00000000001.6.5, , , , 00000000001.7.5, 00000000001.8.5, , ] 2020-10-15T16:42:42.545 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:42.545+0000 7ff4985f60c0 0 bucket checkpoint complete 2020-10-15T16:42:42.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-11 zones={test-zone3, test-zone2} 2020-10-15T16:42:42.585 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:42.591 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:42.591 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:42.592 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:42.592 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:42.592 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:42.598 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:42.598 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:42.599 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:42.599 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:42.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:42.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:42.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:42.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:42.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:43.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:43.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:43.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:43.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:43.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:43.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:43.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:43.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-11 zones={test-zone3, test-zone2} 2020-10-15T16:42:43.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:43.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:43.744 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:43.790 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:43.791 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:44.019 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:44.021+0000 7f1195db00c0 1 bucket sync caught up with source: 2020-10-15T16:42:44.019 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.5.5, , 00000000001.15.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-10-15T16:42:44.020 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.5.5, , 00000000001.15.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-10-15T16:42:44.020 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:44.021+0000 7f1195db00c0 0 bucket checkpoint complete 2020-10-15T16:42:44.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-12 zones={test-zone1, test-zone3} 2020-10-15T16:42:44.049 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:44.054 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:44.055 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:44.055 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:44.055 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:44.056 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:44.060 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:44.061 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:44.061 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:44.061 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:44.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:44.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:44.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:44.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:44.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:44.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:44.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:44.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:44.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:44.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:44.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:45.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:45.141 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-12 zones={test-zone1, test-zone3} 2020-10-15T16:42:45.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:45.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:45.143 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:45.449 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:45.449+0000 7f33d905c0c0 1 bucket sync caught up with source: 2020-10-15T16:42:45.449 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.5.5, , 00000000001.15.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-10-15T16:42:45.449 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000001.5.5, , 00000000001.15.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-10-15T16:42:45.449 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:45.449+0000 7f33d905c0c0 0 bucket checkpoint complete 2020-10-15T16:42:45.481 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-12 zones={test-zone2, test-zone3} 2020-10-15T16:42:45.491 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:45.498 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:45.498 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:45.499 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:45.499 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:45.499 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:45.546 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-15T16:42:45.547 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-15T16:42:45.547 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-15T16:42:45.548 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:45.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:45.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:45.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-15T16:42:45.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:45.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:45.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-15T16:42:46.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:46.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:46.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-15T16:42:46.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:46.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:46.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:46.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-12 zones={test-zone2, test-zone3} 2020-10-15T16:42:46.642 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-10-15T16:42:46.643 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-13 2020-10-15T16:42:46.669 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-14 2020-10-15T16:42:46.726 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-15 2020-10-15T16:42:46.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:46.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:46.924 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:46.964 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:46.964 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:47.144 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:42:47.184 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:42:47.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.185 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:47.185 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:42:47.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780166.765834_106.1", 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:46.765834Z" 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:47.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.187 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:47.187 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:42:47.187 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.187 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:42:47.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:42:47.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:47.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:47.189 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:47.231 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:47.231 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:47.453 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780166.765834_106.1", 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.765834Z", 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:47.462 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:47.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:42:47.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:42:47.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:47.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:47.465 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:47.502 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:47.502 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:47.724 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:42:47.725 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:42:47.725 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:42:47.725 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:42:47.725 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:42:47.726 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:47.726 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:47.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.726 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:42:47.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:42:47.727 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:47.728 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:47.728 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:47.728 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.728 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:42:47.729 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:47.729 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:47.729 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780166.765834_106.1", 2020-10-15T16:42:47.729 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:47.729 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:47.729 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.765834Z", 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:42:47.730 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:47.731 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:47.731 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:47.731 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:47.731 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:47.731 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:47.731 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:42:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:42:47.732 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:42:47.733 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:42:47.733 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:42:47.733 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:42:47.733 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:42:47.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:42:47.734 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:42:47.734 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:42:47.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.734 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:42:47.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:42:47.735 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:42:47.735 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:42:47.735 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:42:47.735 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:42:47.735 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:42:47.736 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:42:47.737 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:42:47.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:47.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:42:47.738 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:42:47.779 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:47.780 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:48.149 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:42:48.149 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:42:48.150 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:42:48.150 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:42:48.150 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:42:48.150 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:42:48.150 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:48.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:48.151 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:42:48.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:48.151 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:42:48.151 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:48.151 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:48.151 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:48.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:48.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:48.153 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:42:48.153 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:48.153 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:48.153 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780166.765834_106.1", 2020-10-15T16:42:48.153 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.765834Z", 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:48.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:48.155 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:42:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:42:48.157 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:42:48.158 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:48.158 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:48.158 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:42:48.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:42:48.158 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:42:48.158 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:42:48.159 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:42:48.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:42:48.159 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:42:48.160 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:42:48.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:48.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:48.161 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:48.246 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:48.246 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:48.495 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:48.493+0000 7fec04c120c0 1 bucket sync caught up with source: 2020-10-15T16:42:48.495 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:48.495 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:48.495 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:48.493+0000 7fec04c120c0 0 bucket checkpoint complete 2020-10-15T16:42:48.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-13 zones={test-zone2, test-zone1} 2020-10-15T16:42:48.537 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:48.586 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:48.587 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:48.593 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:48.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:48.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:48.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:48.917 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-13 zones={test-zone2, test-zone1} 2020-10-15T16:42:48.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:48.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:48.919 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:48.965 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:48.965 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:49.183 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:49.185+0000 7fd846f1d0c0 1 bucket sync caught up with source: 2020-10-15T16:42:49.183 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:49.183 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:49.184 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:49.185+0000 7fd846f1d0c0 0 bucket checkpoint complete 2020-10-15T16:42:49.212 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-13 zones={test-zone3, test-zone1} 2020-10-15T16:42:49.222 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:49.272 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:49.274 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:49.280 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:49.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:49.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:49.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:49.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-13 zones={test-zone3, test-zone1} 2020-10-15T16:42:49.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:49.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:49.607 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:49.912 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:49.913+0000 7fa109c560c0 1 bucket sync caught up with source: 2020-10-15T16:42:49.912 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:49.912 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:49.912 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:49.913+0000 7fa109c560c0 0 bucket checkpoint complete 2020-10-15T16:42:50.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:50.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:50.028 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:50.072 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:50.073 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:50.286 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:50.289+0000 7f7cc5b330c0 1 bucket sync caught up with source: 2020-10-15T16:42:50.287 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:42:50.287 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:42:50.287 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:50.289+0000 7f7cc5b330c0 0 bucket checkpoint complete 2020-10-15T16:42:50.316 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-14 zones={test-zone1, test-zone2} 2020-10-15T16:42:50.327 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:50.375 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:50.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:50.383 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:50.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:50.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:50.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:50.722 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-14 zones={test-zone1, test-zone2} 2020-10-15T16:42:50.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:50.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:50.723 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:50.771 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:50.771 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:51.005 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:51.005+0000 7f8e42fde0c0 1 bucket sync caught up with source: 2020-10-15T16:42:51.005 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:42:51.005 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:42:51.006 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:51.005+0000 7f8e42fde0c0 0 bucket checkpoint complete 2020-10-15T16:42:51.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-14 zones={test-zone3, test-zone2} 2020-10-15T16:42:51.046 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:51.094 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:51.095 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:51.102 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:51.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:51.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:51.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:51.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-14 zones={test-zone3, test-zone2} 2020-10-15T16:42:51.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:51.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:51.431 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:51.476 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:51.477 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:51.715 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:51.717+0000 7f8e256e10c0 1 bucket sync caught up with source: 2020-10-15T16:42:51.715 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2020-10-15T16:42:51.715 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2020-10-15T16:42:51.715 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:42:51.717+0000 7f8e256e10c0 0 bucket checkpoint complete 2020-10-15T16:42:51.741 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-15 zones={test-zone1, test-zone3} 2020-10-15T16:42:51.756 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:51.802 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:51.803 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:51.811 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:51.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:51.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:52.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:52.132 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-15 zones={test-zone1, test-zone3} 2020-10-15T16:42:52.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:52.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:52.134 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:52.181 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:52.181 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:52.421+0000 7f20c0c9a0c0 1 bucket sync caught up with source: 2020-10-15T16:42:52.422 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2020-10-15T16:42:52.422 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2020-10-15T16:42:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:52.421+0000 7f20c0c9a0c0 0 bucket checkpoint complete 2020-10-15T16:42:52.453 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-15 zones={test-zone2, test-zone3} 2020-10-15T16:42:52.463 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:42:52.511 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:52.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:42:52.518 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-15T16:42:52.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:52.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:52.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-15T16:42:52.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-15 zones={test-zone2, test-zone3} 2020-10-15T16:42:52.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:52.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:42:52.864 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:42:52.912 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:42:52.912 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:42:53.163 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:53.165+0000 7f3830dce0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:42:53.163 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:53.163 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:42:58.167 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:42:58.169+0000 7f3830dce0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:42:58.168 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:42:58.168 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:03.171 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:03.173+0000 7f3830dce0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:03.171 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:43:03.171 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:08.175 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:08.177+0000 7f3830dce0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:08.175 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2020-10-15T16:43:08.175 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:13.179 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:13.181+0000 7f3830dce0c0 1 bucket sync caught up with source: 2020-10-15T16:43:13.179 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:13.180 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:13.180 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:13.181+0000 7f3830dce0c0 0 bucket checkpoint complete 2020-10-15T16:43:13.215 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-13 zones={test-zone2, test-zone1} 2020-10-15T16:43:13.226 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:43:13.274 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:43:13.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-13 zones={test-zone2, test-zone1} 2020-10-15T16:43:13.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:13.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:13.338 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:13.381 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:13.382 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:13.607 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:13.609+0000 7fa8bc1360c0 1 bucket sync caught up with source: 2020-10-15T16:43:13.607 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:13.607 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:13.607 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:13.609+0000 7fa8bc1360c0 0 bucket checkpoint complete 2020-10-15T16:43:13.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-13 zones={test-zone3, test-zone1} 2020-10-15T16:43:13.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:43:13.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:43:13.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-13 zones={test-zone3, test-zone1} 2020-10-15T16:43:13.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:13.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:13.758 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:13.808 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:13.809 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:14.065+0000 7fc944cf30c0 1 bucket sync caught up with source: 2020-10-15T16:43:14.064 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:14.065 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2020-10-15T16:43:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:14.065+0000 7fc944cf30c0 0 bucket checkpoint complete 2020-10-15T16:43:14.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:14.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:14.201 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:14.247 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:14.247 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:14.489 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:14.493+0000 7fab981ee0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:14.490 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:43:14.490 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:19.494 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:19.497+0000 7fab981ee0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:19.494 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:43:19.495 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:24.497 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:24.501+0000 7fab981ee0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:24.498 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:43:24.498 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:29.501 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:29.505+0000 7fab981ee0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:29.501 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-15T16:43:29.502 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:34.505 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:34.509+0000 7fab981ee0c0 1 bucket sync caught up with source: 2020-10-15T16:43:34.505 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:34.505 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:34.506 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:34.509+0000 7fab981ee0c0 0 bucket checkpoint complete 2020-10-15T16:43:34.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-14 zones={test-zone1, test-zone2} 2020-10-15T16:43:34.546 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:43:34.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:43:34.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-14 zones={test-zone1, test-zone2} 2020-10-15T16:43:34.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:34.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:34.658 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:34.706 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:34.706 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:34.921 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:34.925+0000 7f8a8e31a0c0 1 bucket sync caught up with source: 2020-10-15T16:43:34.921 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:34.922 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-15T16:43:34.922 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:34.925+0000 7f8a8e31a0c0 0 bucket checkpoint complete 2020-10-15T16:43:34.950 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-14 zones={test-zone3, test-zone2} 2020-10-15T16:43:34.960 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:43:35.010 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:43:35.069 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-14 zones={test-zone3, test-zone2} 2020-10-15T16:43:35.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:35.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:35.089 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:35.130 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:35.131 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:35.349 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:35.353+0000 7f739f9970c0 1 bucket sync caught up with source: 2020-10-15T16:43:35.349 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2020-10-15T16:43:35.350 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2020-10-15T16:43:35.350 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:35.353+0000 7f739f9970c0 0 bucket checkpoint complete 2020-10-15T16:43:35.372 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-15 zones={test-zone1, test-zone3} 2020-10-15T16:43:35.382 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:43:35.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:43:35.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-15 zones={test-zone1, test-zone3} 2020-10-15T16:43:35.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:35.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:35.494 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:35.781 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:35.781+0000 7f626e9c80c0 1 bucket sync caught up with source: 2020-10-15T16:43:35.781 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2020-10-15T16:43:35.781 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2020-10-15T16:43:35.781 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:35.781+0000 7f626e9c80c0 0 bucket checkpoint complete 2020-10-15T16:43:35.813 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-15 zones={test-zone2, test-zone3} 2020-10-15T16:43:35.824 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:43:35.870 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:43:35.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-15 zones={test-zone2, test-zone3} 2020-10-15T16:43:35.933 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-10-15T16:43:35.934 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-16 2020-10-15T16:43:35.962 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-17 2020-10-15T16:43:36.033 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-18 2020-10-15T16:43:36.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.138 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:43:36.179 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:36.179 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:36.360 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:43:36.387 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:43:36.387 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.387 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780216.136755_129.1", 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:43:36.136755Z" 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.388 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:43:36.389 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.390 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:43:36.390 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:43:36.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.392 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:43:36.426 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:36.426 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:43:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:43:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:43:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:43:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:43:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:43:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780216.136755_129.1", 2020-10-15T16:43:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:43:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.136755Z", 2020-10-15T16:43:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:43:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:43:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:43:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:43:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:43:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:43:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:36.691 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:43:36.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:43:36.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:43:36.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.694 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:43:36.734 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:36.735 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:36.978 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:43:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:43:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:43:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:43:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:43:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:43:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:43:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:43:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:43:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780216.136755_129.1", 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.136755Z", 2020-10-15T16:43:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:43:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:43:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:43:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:43:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:43:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:43:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:43:36.987 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:43:36.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:43:36.988 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:43:36.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:43:36.990 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:43:37.031 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:37.031 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:37.302 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:43:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:43:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:43:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:43:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:43:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:43:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780216.136755_129.1", 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.136755Z", 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:43:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:43:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:43:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:43:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:43:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:43:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:43:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:43:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:43:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:43:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:43:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:43:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:43:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:43:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:43:37.313 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:43:37.314 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:43:37.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:37.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:37.364 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:37.413 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:37.413 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:37.641 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:37.641+0000 7fce7f0e20c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:37.642 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T16:43:37.642 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2020-10-15T16:43:42.646 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:42.649+0000 7fce7f0e20c0 1 bucket sync caught up with source: 2020-10-15T16:43:42.646 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-10-15T16:43:42.646 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2020-10-15T16:43:42.647 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:42.649+0000 7fce7f0e20c0 0 bucket checkpoint complete 2020-10-15T16:43:42.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:42.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:42.682 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:42.730 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:42.730 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:42.956 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:42.961+0000 7fe830e3a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:42.956 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-10-15T16:43:42.957 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-10-15T16:43:47.960 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:47.965+0000 7fe830e3a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:47.960 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-10-15T16:43:47.961 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-10-15T16:43:52.964 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:52.969+0000 7fe830e3a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:52.964 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.10.10, , , , , ] 2020-10-15T16:43:52.964 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-10-15T16:43:57.967 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:57.973+0000 7fe830e3a0c0 1 bucket sync caught up with source: 2020-10-15T16:43:57.968 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000006.18.9, , , , , ] 2020-10-15T16:43:57.968 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.16.9, , , , , ] 2020-10-15T16:43:57.968 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:57.973+0000 7fe830e3a0c0 0 bucket checkpoint complete 2020-10-15T16:43:57.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:57.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:58.000 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:58.046 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:58.047 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:58.307 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:58.309+0000 7fb9cee1b0c0 1 bucket sync caught up with source: 2020-10-15T16:43:58.308 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.18.9, , , , , ] 2020-10-15T16:43:58.308 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.18.9, , , , , ] 2020-10-15T16:43:58.308 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:43:58.309+0000 7fb9cee1b0c0 0 bucket checkpoint complete 2020-10-15T16:43:58.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:58.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:43:58.472 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:43:58.519 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:43:58.519 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:43:58.742 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:43:58.745+0000 7f75cd21d0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:43:58.743 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , , , ] 2020-10-15T16:43:58.743 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2020-10-15T16:44:03.746 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:03.749+0000 7f75cd21d0c0 1 bucket sync caught up with source: 2020-10-15T16:44:03.746 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-10-15T16:44:03.747 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2020-10-15T16:44:03.747 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:03.749+0000 7f75cd21d0c0 0 bucket checkpoint complete 2020-10-15T16:44:03.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:03.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:03.783 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:03.829 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:03.829 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:04.037 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:04.041+0000 7f7798df60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:04.037 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-10-15T16:44:04.037 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.27.9, , , , , ] 2020-10-15T16:44:09.042 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:09.045+0000 7f7798df60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:09.042 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-10-15T16:44:09.042 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.27.9, , , , , ] 2020-10-15T16:44:14.044 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:14.049+0000 7f7798df60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:14.045 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-10-15T16:44:14.045 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.27.9, , , , , ] 2020-10-15T16:44:19.049 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:19.053+0000 7f7798df60c0 1 bucket sync caught up with source: 2020-10-15T16:44:19.050 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000006.29.9, , , , , ] 2020-10-15T16:44:19.050 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.27.9, , , , , ] 2020-10-15T16:44:19.050 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:19.053+0000 7f7798df60c0 0 bucket checkpoint complete 2020-10-15T16:44:19.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:19.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:19.123 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:19.169 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:19.169 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:19.390 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:19.392+0000 7fe6dc44f0c0 1 bucket sync caught up with source: 2020-10-15T16:44:19.390 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-15T16:44:19.390 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000002.19.10, , , , , ] 2020-10-15T16:44:19.391 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:19.392+0000 7fe6dc44f0c0 0 bucket checkpoint complete 2020-10-15T16:44:19.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:19.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:19.413 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:19.466 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:19.466 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:19.690 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:19.693+0000 7f4a6ba220c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:19.691 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-15T16:44:19.691 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2020-10-15T16:44:24.695 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:24.697+0000 7f4a6ba220c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:24.695 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-15T16:44:24.695 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2020-10-15T16:44:29.700 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:29.704+0000 7f4a6ba220c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:29.701 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-15T16:44:29.701 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2020-10-15T16:44:34.706 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:34.707+0000 7f4a6ba220c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:34.706 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-15T16:44:34.707 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2020-10-15T16:44:39.711 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:39.710+0000 7f4a6ba220c0 1 bucket sync caught up with source: 2020-10-15T16:44:39.711 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000005.25.9, , , , , ] 2020-10-15T16:44:39.711 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2020-10-15T16:44:39.712 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:39.710+0000 7f4a6ba220c0 0 bucket checkpoint complete 2020-10-15T16:44:41.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:41.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:41.450 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:41.497 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:41.498 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:41.699 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:41.698+0000 7fcba60780c0 1 bucket sync caught up with source: 2020-10-15T16:44:41.699 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.29.5, , 00000000003.13.9, , 00000000006.18.9, , , , , ] 2020-10-15T16:44:41.699 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.29.5, , 00000000003.13.9, , 00000000006.18.9, , , , , ] 2020-10-15T16:44:41.699 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:41.698+0000 7fcba60780c0 0 bucket checkpoint complete 2020-10-15T16:44:41.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:41.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:41.732 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:41.782 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:41.782 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:41.987 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:41.984+0000 7f730cf960c0 1 bucket sync caught up with source: 2020-10-15T16:44:41.988 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000010.29.5, , 00000000003.13.9, , 00000000006.18.9, , , , , ] 2020-10-15T16:44:41.988 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000010.29.5, , 00000000003.13.9, , 00000000006.18.9, , , , , ] 2020-10-15T16:44:41.988 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:41.984+0000 7f730cf960c0 0 bucket checkpoint complete 2020-10-15T16:44:42.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:42.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:42.014 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:42.063 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:42.063 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:42.292 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:42.289+0000 7f120c51c0c0 1 bucket sync caught up with source: 2020-10-15T16:44:42.292 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.29.5, , 00000000003.13.9, , 00000000006.18.9, , , , , ] 2020-10-15T16:44:42.292 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.29.5, , 00000000003.13.9, , 00000000006.18.9, , , , , ] 2020-10-15T16:44:42.292 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:44:42.289+0000 7f120c51c0c0 0 bucket checkpoint complete 2020-10-15T16:44:42.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:42.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:44:42.337 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:44:42.383 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:44:42.384 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:44:42.614 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:42.612+0000 7fba130cc0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:42.614 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.19.10, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:42.614 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:47.622 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:47.615+0000 7fba130cc0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:47.623 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.19.10, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:47.623 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:52.624 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:52.622+0000 7fba130cc0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:52.624 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.19.10, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:52.624 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:57.628 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:44:57.626+0000 7fba130cc0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:44:57.629 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000002.19.10, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:44:57.629 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:45:02.633 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:02.629+0000 7fba130cc0c0 1 bucket sync caught up with source: 2020-10-15T16:45:02.633 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:02.633 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000004.20.9, , , , , ] 2020-10-15T16:45:02.633 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:02.629+0000 7fba130cc0c0 0 bucket checkpoint complete 2020-10-15T16:45:02.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:02.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:02.665 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:02.752 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:02.753 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:02.981 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:02.977+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:02.981 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:02.982 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:07.985 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:07.981+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:07.985 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:07.986 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:12.989 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:12.985+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:12.990 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:12.990 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:17.994 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:17.988+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:17.994 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:17.994 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:22.996 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:22.992+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:22.996 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:22.997 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:28.000 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:27.995+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:28.001 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:28.001 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:33.004 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:33.003+0000 7f2c98bfd0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:33.005 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000006.72.10, , , , , ] 2020-10-15T16:45:33.005 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:38.009 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:38.006+0000 7f2c98bfd0c0 1 bucket sync caught up with source: 2020-10-15T16:45:38.009 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:38.009 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000010.35.5, , 00000000007.74.9, , , , , ] 2020-10-15T16:45:38.010 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:38.006+0000 7f2c98bfd0c0 0 bucket checkpoint complete 2020-10-15T16:45:38.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:38.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:38.044 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:38.087 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:38.088 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:38.310 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:38.306+0000 7f900846b0c0 1 bucket sync caught up with source: 2020-10-15T16:45:38.311 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.21.9, , 00000000003.17.9, , 00000000014.39.5, , , , , ] 2020-10-15T16:45:38.311 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.21.9, , 00000000003.17.9, , 00000000014.39.5, , , , , ] 2020-10-15T16:45:38.311 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:38.306+0000 7f900846b0c0 0 bucket checkpoint complete 2020-10-15T16:45:38.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:38.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:38.338 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:38.386 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:38.386 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:38.636+0000 7f6d047b90c0 1 bucket sync caught up with source: 2020-10-15T16:45:38.638 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000003.21.9, , 00000000003.17.9, , 00000000014.39.5, , , , , ] 2020-10-15T16:45:38.638 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000003.21.9, , 00000000003.17.9, , 00000000014.39.5, , , , , ] 2020-10-15T16:45:38.638 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:38.636+0000 7f6d047b90c0 0 bucket checkpoint complete 2020-10-15T16:45:38.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-16 zones={test-zone2, test-zone1} 2020-10-15T16:45:38.678 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:45:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:38.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:38.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:38.728 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:45:38.735 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:38.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:38.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:38.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:38.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:38.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:38.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:38.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:38.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:39.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:39.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:39.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:39.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:39.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:39.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:39.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:39.713 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-16 zones={test-zone2, test-zone1} 2020-10-15T16:45:39.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-16 zones={test-zone3, test-zone1} 2020-10-15T16:45:39.723 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:45:39.728 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:39.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:39.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:39.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:39.729 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:45:39.733 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:39.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:39.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:39.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:39.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:39.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:39.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:39.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:39.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:40.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:40.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:40.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:40.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:40.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:40.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:40.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:40.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-16 zones={test-zone3, test-zone1} 2020-10-15T16:45:40.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-16 zones={test-zone3, test-zone2} 2020-10-15T16:45:40.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:45:40.730 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:40.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:40.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:40.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:40.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:45:40.738 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:40.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:40.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:40.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:40.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:40.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:40.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:40.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:40.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:41.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:41.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:41.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:41.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:41.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:41.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:41.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:41.722 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-16 zones={test-zone3, test-zone2} 2020-10-15T16:45:41.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:41.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:41.723 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:42.025 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:42.019+0000 7f0ff45780c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:42.026 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.22.5, , 00000000003.24.9, , 00000000006.45.10, , , , , ] 2020-10-15T16:45:42.026 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:47.030 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:47.023+0000 7f0ff45780c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:47.030 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.22.5, , 00000000003.24.9, , 00000000006.45.10, , , , , ] 2020-10-15T16:45:47.030 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:52.034 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:52.027+0000 7f0ff45780c0 1 waiting for incremental sync to catch up: 2020-10-15T16:45:52.034 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.22.5, , 00000000003.24.9, , 00000000006.45.10, , , , , ] 2020-10-15T16:45:52.034 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.038 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:57.034+0000 7f0ff45780c0 1 bucket sync caught up with source: 2020-10-15T16:45:57.039 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.039 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.039 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:57.034+0000 7f0ff45780c0 0 bucket checkpoint complete 2020-10-15T16:45:57.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:57.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:57.074 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:57.119 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:57.119 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:57.360 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:57.353+0000 7f16a18a30c0 1 bucket sync caught up with source: 2020-10-15T16:45:57.361 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.361 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.361 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:57.353+0000 7f16a18a30c0 0 bucket checkpoint complete 2020-10-15T16:45:57.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:57.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:57.387 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:57.693 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:57.686+0000 7f43ccc0d0c0 1 bucket sync caught up with source: 2020-10-15T16:45:57.694 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.694 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.22.5, , 00000000003.24.9, , 00000000007.47.9, , , , , ] 2020-10-15T16:45:57.694 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:57.686+0000 7f43ccc0d0c0 0 bucket checkpoint complete 2020-10-15T16:45:57.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:57.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:57.810 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:57.858 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:57.858 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:58.081 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:58.077+0000 7f98467b70c0 1 bucket sync caught up with source: 2020-10-15T16:45:58.081 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.12.9, , 00000000010.40.5, , 00000000009.41.9, , , , , ] 2020-10-15T16:45:58.081 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.12.9, , 00000000010.40.5, , 00000000009.41.9, , , , , ] 2020-10-15T16:45:58.081 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:58.077+0000 7f98467b70c0 0 bucket checkpoint complete 2020-10-15T16:45:58.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:58.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:58.111 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:58.157 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:58.157 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:58.375 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:58.369+0000 7f5a5d3e60c0 1 bucket sync caught up with source: 2020-10-15T16:45:58.375 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.12.9, , 00000000010.40.5, , 00000000009.41.9, , , , , ] 2020-10-15T16:45:58.375 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.12.9, , 00000000010.40.5, , 00000000009.41.9, , , , , ] 2020-10-15T16:45:58.375 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:58.369+0000 7f5a5d3e60c0 0 bucket checkpoint complete 2020-10-15T16:45:58.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:58.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:58.406 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:58.452 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:58.453 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:58.681 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:58.677+0000 7fe7e16190c0 1 bucket sync caught up with source: 2020-10-15T16:45:58.681 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.12.9, , 00000000003.45.9, , 00000000014.44.5, , , , , ] 2020-10-15T16:45:58.682 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.12.9, , 00000000003.45.9, , 00000000014.44.5, , , , , ] 2020-10-15T16:45:58.682 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:45:58.677+0000 7fe7e16190c0 0 bucket checkpoint complete 2020-10-15T16:45:58.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:58.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:45:58.709 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:45:58.758 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:45:58.758 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:45:59.017 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:59.010+0000 7efe0f5920c0 1 bucket sync caught up with source: 2020-10-15T16:45:59.017 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000003.12.9, , 00000000003.45.9, , 00000000014.44.5, , , , , ] 2020-10-15T16:45:59.017 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000003.12.9, , 00000000003.45.9, , 00000000014.44.5, , , , , ] 2020-10-15T16:45:59.018 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:45:59.010+0000 7efe0f5920c0 0 bucket checkpoint complete 2020-10-15T16:45:59.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-17 zones={test-zone2, test-zone1} 2020-10-15T16:45:59.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:45:59.070 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:59.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:59.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:59.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:59.072 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:45:59.079 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:45:59.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:45:59.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:45:59.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:45:59.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:59.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:59.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:45:59.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:59.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:59.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:45:59.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:59.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:59.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:45:59.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:59.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:45:59.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:00.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-17 zones={test-zone2, test-zone1} 2020-10-15T16:46:00.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-17 zones={test-zone3, test-zone1} 2020-10-15T16:46:00.074 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:00.078 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:00.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:00.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:00.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:00.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:00.084 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:00.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:00.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:00.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:00.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:00.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:00.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:00.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:00.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:00.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:00.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:00.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:00.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:00.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:00.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:00.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:01.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-17 zones={test-zone3, test-zone1} 2020-10-15T16:46:01.055 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-17 zones={test-zone3, test-zone2} 2020-10-15T16:46:01.065 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:01.070 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:01.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:01.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:01.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:01.071 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:01.077 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:01.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:01.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:01.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:01.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:01.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:01.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:01.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:01.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:01.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:01.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:01.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:01.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:01.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:01.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:01.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:02.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-17 zones={test-zone3, test-zone2} 2020-10-15T16:46:02.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:02.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:02.051 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:02.096 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:02.096 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:02.334 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:02.326+0000 7fdffaff20c0 1 bucket sync caught up with source: 2020-10-15T16:46:02.335 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.37.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-10-15T16:46:02.335 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.37.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-10-15T16:46:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:02.330+0000 7fdffaff20c0 0 bucket checkpoint complete 2020-10-15T16:46:02.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:02.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:02.370 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:02.416 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:02.417 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:02.646 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:02.641+0000 7fed51e320c0 1 bucket sync caught up with source: 2020-10-15T16:46:02.647 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000010.37.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-10-15T16:46:02.647 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000010.37.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-10-15T16:46:02.647 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:02.641+0000 7fed51e320c0 0 bucket checkpoint complete 2020-10-15T16:46:02.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:02.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:02.673 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:02.720 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:02.982 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:02.974+0000 7f71819cc0c0 1 bucket sync caught up with source: 2020-10-15T16:46:02.982 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000010.37.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-10-15T16:46:02.982 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000010.37.5, , 00000000003.30.9, , 00000000007.21.9, , , , , ] 2020-10-15T16:46:02.982 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:02.974+0000 7f71819cc0c0 0 bucket checkpoint complete 2020-10-15T16:46:03.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:03.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:03.094 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:03.138 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:03.139 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:03.375 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:03.369+0000 7f6aac3770c0 1 bucket sync caught up with source: 2020-10-15T16:46:03.375 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.35.9, , 00000000010.63.5, , 00000000007.23.9, , , , , ] 2020-10-15T16:46:03.375 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.35.9, , 00000000010.63.5, , 00000000007.23.9, , , , , ] 2020-10-15T16:46:03.375 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:03.369+0000 7f6aac3770c0 0 bucket checkpoint complete 2020-10-15T16:46:03.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:03.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:03.406 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:03.494 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:03.494 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:03.725 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:03.721+0000 7f8b4f4de0c0 1 bucket sync caught up with source: 2020-10-15T16:46:03.726 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.35.9, , 00000000010.63.5, , 00000000007.23.9, , , , , ] 2020-10-15T16:46:03.726 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.35.9, , 00000000010.63.5, , 00000000007.23.9, , , , , ] 2020-10-15T16:46:03.726 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:03.721+0000 7f8b4f4de0c0 0 bucket checkpoint complete 2020-10-15T16:46:03.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:03.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:03.755 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:03.798 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:03.798 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:04.018 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:04.013+0000 7fa9758820c0 1 bucket sync caught up with source: 2020-10-15T16:46:04.019 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000003.50.9, , 00000000003.51.9, , 00000000016.47.5, , , , , ] 2020-10-15T16:46:04.019 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000003.50.9, , 00000000003.51.9, , 00000000016.47.5, , , , , ] 2020-10-15T16:46:04.019 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:04.013+0000 7fa9758820c0 0 bucket checkpoint complete 2020-10-15T16:46:04.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:04.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:04.045 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:04.091 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:04.092 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:04.328 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:04.322+0000 7fdd9b33e0c0 1 bucket sync caught up with source: 2020-10-15T16:46:04.328 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000003.50.9, , 00000000003.51.9, , 00000000016.47.5, , , , , ] 2020-10-15T16:46:04.329 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000003.50.9, , 00000000003.51.9, , 00000000016.47.5, , , , , ] 2020-10-15T16:46:04.329 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:04.322+0000 7fdd9b33e0c0 0 bucket checkpoint complete 2020-10-15T16:46:04.360 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-18 zones={test-zone2, test-zone1} 2020-10-15T16:46:04.371 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:04.377 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:04.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:04.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:04.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:04.379 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:04.385 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:04.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:04.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:04.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:04.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:04.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:04.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:04.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:04.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:04.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:04.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:04.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:05.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:05.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:05.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:05.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:05.401 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-18 zones={test-zone2, test-zone1} 2020-10-15T16:46:05.402 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-18 zones={test-zone3, test-zone1} 2020-10-15T16:46:05.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:05.415 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:05.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:05.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:05.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:05.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:05.421 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:05.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:05.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:05.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:05.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:05.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:05.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:05.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:05.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:05.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:05.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:05.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:06.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:06.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:06.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:06.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:06.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-18 zones={test-zone3, test-zone1} 2020-10-15T16:46:06.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-18 zones={test-zone3, test-zone2} 2020-10-15T16:46:06.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:06.403 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:06.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:06.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:06.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:06.405 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:06.412 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:06.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:06.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:06.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:06.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:06.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:06.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:06.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:06.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:06.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:06.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:06.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:07.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:07.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:07.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:07.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:07.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-18 zones={test-zone3, test-zone2} 2020-10-15T16:46:08.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:08.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:08.072 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:08.324 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:08.318+0000 7fcace10c0c0 1 bucket sync caught up with source: 2020-10-15T16:46:08.324 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.36.5, , 00000000004.16.5, , 00000000010.39.5, , , , , ] 2020-10-15T16:46:08.325 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.36.5, , 00000000004.16.5, , 00000000010.39.5, , , , , ] 2020-10-15T16:46:08.325 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:08.318+0000 7fcace10c0c0 0 bucket checkpoint complete 2020-10-15T16:46:08.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:08.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:08.357 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:08.406 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:08.407 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:08.603 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:08.597+0000 7fe30f4a80c0 1 bucket sync caught up with source: 2020-10-15T16:46:08.604 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000011.36.5, , 00000000004.16.5, , 00000000010.39.5, , , , , ] 2020-10-15T16:46:08.604 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000011.36.5, , 00000000004.16.5, , 00000000010.39.5, , , , , ] 2020-10-15T16:46:08.604 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:08.597+0000 7fe30f4a80c0 0 bucket checkpoint complete 2020-10-15T16:46:08.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:08.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:08.629 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:08.678 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:08.678 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:08.929 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:08.922+0000 7f58e8e5e0c0 1 bucket sync caught up with source: 2020-10-15T16:46:08.930 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.36.5, , 00000000004.16.5, , 00000000010.39.5, , , , , ] 2020-10-15T16:46:08.930 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.36.5, , 00000000004.16.5, , 00000000010.39.5, , , , , ] 2020-10-15T16:46:08.930 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:08.922+0000 7f58e8e5e0c0 0 bucket checkpoint complete 2020-10-15T16:46:09.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:09.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:09.054 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:09.100 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:09.100 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:09.332 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:09.325+0000 7f7dbecce0c0 1 bucket sync caught up with source: 2020-10-15T16:46:09.332 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.24.5, , 00000000011.42.5, , 00000000008.79.5, , , , , ] 2020-10-15T16:46:09.332 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.24.5, , 00000000011.42.5, , 00000000008.79.5, , , , , ] 2020-10-15T16:46:09.333 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:09.325+0000 7f7dbecce0c0 0 bucket checkpoint complete 2020-10-15T16:46:09.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:09.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:09.363 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:09.449 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:09.450 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:09.691 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:09.685+0000 7f433bf3b0c0 1 bucket sync caught up with source: 2020-10-15T16:46:09.691 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.24.5, , 00000000011.42.5, , 00000000008.79.5, , , , , ] 2020-10-15T16:46:09.691 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.24.5, , 00000000011.42.5, , 00000000008.79.5, , , , , ] 2020-10-15T16:46:09.691 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:09.685+0000 7f433bf3b0c0 0 bucket checkpoint complete 2020-10-15T16:46:09.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:09.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:09.723 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:09.765 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:09.766 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:09.991 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:09.985+0000 7f873c4110c0 1 bucket sync caught up with source: 2020-10-15T16:46:09.992 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.24.5, , 00000000004.20.5, , 00000000015.54.5, , , , , ] 2020-10-15T16:46:09.992 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.24.5, , 00000000004.20.5, , 00000000015.54.5, , , , , ] 2020-10-15T16:46:09.992 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:09.985+0000 7f873c4110c0 0 bucket checkpoint complete 2020-10-15T16:46:10.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:10.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:10.015 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:10.065 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:10.066 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:10.322 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:10.313+0000 7f29961600c0 1 bucket sync caught up with source: 2020-10-15T16:46:10.322 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000004.24.5, , 00000000004.20.5, , 00000000015.54.5, , , , , ] 2020-10-15T16:46:10.322 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000004.24.5, , 00000000004.20.5, , 00000000015.54.5, , , , , ] 2020-10-15T16:46:10.322 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:10.313+0000 7f29961600c0 0 bucket checkpoint complete 2020-10-15T16:46:10.354 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-16 zones={test-zone2, test-zone1} 2020-10-15T16:46:10.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:10.372 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:10.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:10.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:10.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:10.373 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:10.379 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:10.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:10.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:10.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:10.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:10.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:10.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:10.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:10.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:10.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:10.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:10.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:11.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:11.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:11.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:11.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:11.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-16 zones={test-zone2, test-zone1} 2020-10-15T16:46:11.406 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-16 zones={test-zone3, test-zone1} 2020-10-15T16:46:11.414 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:11.419 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:11.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:11.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:11.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:11.420 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:11.424 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:11.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:11.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:11.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:11.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:11.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:11.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:11.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:11.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:11.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:11.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:11.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:12.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:12.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:12.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:12.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:12.453 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-16 zones={test-zone3, test-zone1} 2020-10-15T16:46:12.454 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-16 zones={test-zone3, test-zone2} 2020-10-15T16:46:12.465 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:12.472 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:12.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:12.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:12.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:12.473 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:12.482 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:12.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:12.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:12.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:12.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:12.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:12.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:12.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:12.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:12.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:12.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:13.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:13.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:13.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:13.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:13.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:13.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-16 zones={test-zone3, test-zone2} 2020-10-15T16:46:13.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:13.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:13.508 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:13.555 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:13.556 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:13.801 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:13.793+0000 7f547c6300c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:13.802 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000007.47.9, , , , , ] 2020-10-15T16:46:13.802 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:18.806 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:18.797+0000 7f547c6300c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:18.806 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000007.47.9, , , , , ] 2020-10-15T16:46:18.806 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:23.810 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:23.801+0000 7f547c6300c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:23.811 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000007.47.9, , , , , ] 2020-10-15T16:46:23.811 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:28.815 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:28.805+0000 7f547c6300c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:28.815 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000007.47.9, , , , , ] 2020-10-15T16:46:28.815 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:33.819 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:33.809+0000 7f547c6300c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:33.819 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000007.47.9, , , , , ] 2020-10-15T16:46:33.819 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:38.822 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:38.813+0000 7f547c6300c0 1 bucket sync caught up with source: 2020-10-15T16:46:38.823 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:38.823 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:38.823 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:38.813+0000 7f547c6300c0 0 bucket checkpoint complete 2020-10-15T16:46:38.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:38.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:38.858 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:38.904 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:38.904 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:39.129 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:39.124+0000 7f21f2d4c0c0 1 bucket sync caught up with source: 2020-10-15T16:46:39.130 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:39.130 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:39.130 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:39.124+0000 7f21f2d4c0c0 0 bucket checkpoint complete 2020-10-15T16:46:39.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:39.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:39.156 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:39.465 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:39.457+0000 7f12a131b0c0 1 bucket sync caught up with source: 2020-10-15T16:46:39.465 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:39.465 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.29.5, , 00000000004.41.5, , 00000000008.52.5, , , , , ] 2020-10-15T16:46:39.466 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:39.457+0000 7f12a131b0c0 0 bucket checkpoint complete 2020-10-15T16:46:39.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:39.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:39.577 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:39.625 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:39.626 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:39.859 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:39.852+0000 7f57e4ff40c0 1 bucket sync caught up with source: 2020-10-15T16:46:39.859 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.15.5, , 00000000011.77.5, , 00000000010.45.5, , , , , ] 2020-10-15T16:46:39.859 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.15.5, , 00000000011.77.5, , 00000000010.45.5, , , , , ] 2020-10-15T16:46:39.859 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:39.852+0000 7f57e4ff40c0 0 bucket checkpoint complete 2020-10-15T16:46:39.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:39.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:39.887 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:39.934 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:39.935 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:40.161 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:40.156+0000 7f50877530c0 1 bucket sync caught up with source: 2020-10-15T16:46:40.161 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.15.5, , 00000000011.77.5, , 00000000010.45.5, , , , , ] 2020-10-15T16:46:40.161 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.15.5, , 00000000011.77.5, , 00000000010.45.5, , , , , ] 2020-10-15T16:46:40.162 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:40.156+0000 7f50877530c0 0 bucket checkpoint complete 2020-10-15T16:46:40.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:40.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:40.195 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:40.237 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:40.238 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:40.462 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:40.456+0000 7fecdef220c0 1 bucket sync caught up with source: 2020-10-15T16:46:40.462 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.15.5, , 00000000004.56.5, , 00000000015.53.5, , , , , ] 2020-10-15T16:46:40.462 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.15.5, , 00000000004.56.5, , 00000000015.53.5, , , , , ] 2020-10-15T16:46:40.462 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:40.456+0000 7fecdef220c0 0 bucket checkpoint complete 2020-10-15T16:46:40.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:40.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:40.489 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:40.536 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:40.536 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:40.808 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:40.801+0000 7ff6706fe0c0 1 bucket sync caught up with source: 2020-10-15T16:46:40.808 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000004.15.5, , 00000000004.56.5, , 00000000015.53.5, , , , , ] 2020-10-15T16:46:40.808 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000004.15.5, , 00000000004.56.5, , 00000000015.53.5, , , , , ] 2020-10-15T16:46:40.808 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:40.801+0000 7ff6706fe0c0 0 bucket checkpoint complete 2020-10-15T16:46:40.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-17 zones={test-zone2, test-zone1} 2020-10-15T16:46:40.851 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:40.859 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:40.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:40.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:40.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:40.860 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:40.867 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:40.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:40.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:40.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:40.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:40.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:40.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:41.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:41.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:41.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:41.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:41.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:41.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:41.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:41.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:41.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:41.850 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-17 zones={test-zone2, test-zone1} 2020-10-15T16:46:41.851 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-17 zones={test-zone3, test-zone1} 2020-10-15T16:46:41.860 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:41.866 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:41.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:41.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:41.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:41.868 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:41.874 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:41.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:41.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:41.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:41.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:41.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:41.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:42.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:42.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:42.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:42.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:42.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:42.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:42.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:42.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:42.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:42.862 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-17 zones={test-zone3, test-zone1} 2020-10-15T16:46:42.862 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-17 zones={test-zone3, test-zone2} 2020-10-15T16:46:42.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:42.878 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:42.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:42.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:42.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:42.879 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:42.886 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:42.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:42.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:42.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:42.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:42.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:42.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:43.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:43.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:43.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:43.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:43.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:43.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:43.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:43.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:43.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:43.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-17 zones={test-zone3, test-zone2} 2020-10-15T16:46:43.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:43.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:43.872 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:43.917 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:43.917 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:44.163 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:44.157+0000 7f3e816260c0 1 bucket sync caught up with source: 2020-10-15T16:46:44.164 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.50.5, , 00000000004.43.5, , 00000000008.24.5, , , , , ] 2020-10-15T16:46:44.164 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.50.5, , 00000000004.43.5, , 00000000008.24.5, , , , , ] 2020-10-15T16:46:44.164 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:44.157+0000 7f3e816260c0 0 bucket checkpoint complete 2020-10-15T16:46:44.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:44.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:44.192 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:44.238 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:44.239 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:44.459 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:44.452+0000 7fa703a730c0 1 bucket sync caught up with source: 2020-10-15T16:46:44.459 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000011.50.5, , 00000000004.43.5, , 00000000008.24.5, , , , , ] 2020-10-15T16:46:44.460 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000011.50.5, , 00000000004.43.5, , 00000000008.24.5, , , , , ] 2020-10-15T16:46:44.460 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:44.452+0000 7fa703a730c0 0 bucket checkpoint complete 2020-10-15T16:46:44.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:44.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:44.484 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:44.532 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:44.532 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:44.784 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:44.777+0000 7f94d2f970c0 1 bucket sync caught up with source: 2020-10-15T16:46:44.785 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000011.50.5, , 00000000004.43.5, , 00000000008.24.5, , , , , ] 2020-10-15T16:46:44.785 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000011.50.5, , 00000000004.43.5, , 00000000008.24.5, , , , , ] 2020-10-15T16:46:44.785 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:44.777+0000 7f94d2f970c0 0 bucket checkpoint complete 2020-10-15T16:46:44.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:44.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:44.901 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:44.948 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:44.948 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:45.175 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:45.167+0000 7f502adb30c0 1 bucket sync caught up with source: 2020-10-15T16:46:45.175 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.46.5, , 00000000011.81.5, , 00000000008.26.5, , , , , ] 2020-10-15T16:46:45.175 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.46.5, , 00000000011.81.5, , 00000000008.26.5, , , , , ] 2020-10-15T16:46:45.175 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:45.167+0000 7f502adb30c0 0 bucket checkpoint complete 2020-10-15T16:46:45.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:45.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:45.209 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:45.254 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:45.255 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:45.493 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:45.487+0000 7f3cbedee0c0 1 bucket sync caught up with source: 2020-10-15T16:46:45.494 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.46.5, , 00000000011.81.5, , 00000000008.26.5, , , , , ] 2020-10-15T16:46:45.494 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.46.5, , 00000000011.81.5, , 00000000008.26.5, , , , , ] 2020-10-15T16:46:45.494 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:45.487+0000 7f3cbedee0c0 0 bucket checkpoint complete 2020-10-15T16:46:45.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:45.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:45.524 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:45.571 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:45.572 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:45.803 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:45.795+0000 7fd249dd90c0 1 bucket sync caught up with source: 2020-10-15T16:46:45.803 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, 00000000004.55.5, , 00000000004.58.5, , 00000000017.50.5, , , , , ] 2020-10-15T16:46:45.803 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, 00000000004.55.5, , 00000000004.58.5, , 00000000017.50.5, , , , , ] 2020-10-15T16:46:45.804 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:45.795+0000 7fd249dd90c0 0 bucket checkpoint complete 2020-10-15T16:46:45.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:45.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:45.830 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:45.879 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:45.879 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:46.139 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:46.129+0000 7f597ddf90c0 1 bucket sync caught up with source: 2020-10-15T16:46:46.139 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, 00000000004.55.5, , 00000000004.58.5, , 00000000017.50.5, , , , , ] 2020-10-15T16:46:46.140 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, 00000000004.55.5, , 00000000004.58.5, , 00000000017.50.5, , , , , ] 2020-10-15T16:46:46.140 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:46.129+0000 7f597ddf90c0 0 bucket checkpoint complete 2020-10-15T16:46:46.171 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-18 zones={test-zone2, test-zone1} 2020-10-15T16:46:46.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:46.187 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:46.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:46.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:46.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:46.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:46.198 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:46.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:46.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:46.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:46.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:46.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:46.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:46.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:46.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:46.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:46.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:46.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:46.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:46.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:47.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:47.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:47.230 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-18 zones={test-zone2, test-zone1} 2020-10-15T16:46:47.231 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-18 zones={test-zone3, test-zone1} 2020-10-15T16:46:47.239 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:47.243 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:47.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:47.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:47.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:47.245 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:47.249 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:47.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:47.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:47.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:47.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:47.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:47.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:47.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:47.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:47.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:47.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:47.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:47.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:47.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:47.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:48.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:48.207 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-18 zones={test-zone3, test-zone1} 2020-10-15T16:46:48.208 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-18 zones={test-zone3, test-zone2} 2020-10-15T16:46:48.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:46:48.223 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:48.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:48.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:48.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:48.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:46:48.231 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:46:48.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-15T16:46:48.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-15T16:46:48.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-15T16:46:48.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:48.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:48.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:46:48.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:48.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:48.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-15T16:46:48.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:48.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:48.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-15T16:46:48.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:49.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:49.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-15T16:46:49.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-18 zones={test-zone3, test-zone2} 2020-10-15T16:46:49.218 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-10-15T16:46:49.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:49.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:49.269 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:46:49.306 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:49.307 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:49.524 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:46:49.555 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:46:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:46:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:46:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:49.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780409.263968_150.1", 2020-10-15T16:46:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:49.263968Z" 2020-10-15T16:46:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:46:49.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:46:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:46:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:46:49.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:49.559 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:46:49.560 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:46:49.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:49.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:49.561 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:46:49.602 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:49.602 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:49.870 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:46:49.871 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:46:49.871 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:46:49.871 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:46:49.871 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:46:49.871 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:46:49.871 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:49.872 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:49.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780409.263968_150.1", 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:49.874 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.263968Z", 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:46:49.875 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:46:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:49.877 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:49.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:49.879 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:46:49.880 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:46:49.881 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:46:49.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:49.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:49.882 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:46:49.922 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:49.923 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:50.173 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:46:50.174 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:46:50.174 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:46:50.174 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:46:50.174 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:46:50.174 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:46:50.174 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:46:50.175 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:50.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780409.263968_150.1", 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:46:50.177 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:46:50.178 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.263968Z", 2020-10-15T16:46:50.178 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:46:50.178 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:50.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:50.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:50.178 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:46:50.179 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:46:50.180 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:50.181 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:46:50.182 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:46:50.183 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:46:50.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:46:50.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:50.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:46:50.185 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:46:50.225 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:50.225 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:50.582 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:46:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:46:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:46:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:46:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:46:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:46:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:46:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:46:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780409.263968_150.1", 2020-10-15T16:46:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:46:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.263968Z", 2020-10-15T16:46:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:46:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:46:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:46:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:46:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:46:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:46:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:46:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:46:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:46:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:46:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:46:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:46:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:46:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:46:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:46:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:46:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:46:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:46:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:46:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:46:50.600 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:46:50.601 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:46:50.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:50.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:50.656 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:50.704 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:50.705 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:50.942 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:50.933+0000 7f7349aa20c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:50.943 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T16:46:50.943 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:46:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:55.937+0000 7f7349aa20c0 1 bucket sync caught up with source: 2020-10-15T16:46:55.947 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:46:55.947 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:46:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:46:55.937+0000 7f7349aa20c0 0 bucket checkpoint complete 2020-10-15T16:46:55.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:55.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:46:55.980 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:46:56.036 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:46:56.037 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:46:56.236 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:46:56.227+0000 7f2777a9a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:46:56.237 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:46:56.237 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:01.240 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:01.231+0000 7f2777a9a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:01.240 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:47:01.241 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:06.244 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:06.235+0000 7f2777a9a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:06.244 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:47:06.244 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:11.248 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:11.243+0000 7f2777a9a0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:11.248 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2020-10-15T16:47:11.248 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:16.251 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:16.247+0000 7f2777a9a0c0 1 bucket sync caught up with source: 2020-10-15T16:47:16.252 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:16.252 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:16.252 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:16.247+0000 7f2777a9a0c0 0 bucket checkpoint complete 2020-10-15T16:47:16.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:16.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:16.283 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:47:16.332 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:47:16.332 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:47:16.600 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:16.593+0000 7fd4657a60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:16.601 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:16.601 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:21.605 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:21.597+0000 7fd4657a60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:21.605 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:21.605 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:26.608 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:26.601+0000 7fd4657a60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:26.609 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:26.609 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:31.612 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:31.605+0000 7fd4657a60c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:31.613 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2020-10-15T16:47:31.613 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:36.617 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:36.609+0000 7fd4657a60c0 1 bucket sync caught up with source: 2020-10-15T16:47:36.617 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:36.617 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2020-10-15T16:47:36.617 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:36.609+0000 7fd4657a60c0 0 bucket checkpoint complete 2020-10-15T16:47:36.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:36.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:36.829 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:47:36.874 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:47:36.874 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:47:37.097 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:37.094+0000 7f6b3de100c0 1 bucket sync caught up with source: 2020-10-15T16:47:37.097 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2020-10-15T16:47:37.097 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-10-15T16:47:37.097 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:37.094+0000 7f6b3de100c0 0 bucket checkpoint complete 2020-10-15T16:47:37.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:37.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:37.124 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:47:37.169 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:47:37.169 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:47:37.400 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:37.394+0000 7f362ea350c0 1 bucket sync caught up with source: 2020-10-15T16:47:37.401 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2020-10-15T16:47:37.401 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-10-15T16:47:37.401 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:37.394+0000 7f362ea350c0 0 bucket checkpoint complete 2020-10-15T16:47:37.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:37.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:37.432 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:47:37.479 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:47:37.479 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:47:37.719 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:37.714+0000 7f44cb7c40c0 1 bucket sync caught up with source: 2020-10-15T16:47:37.720 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.17.9, , , , , ] 2020-10-15T16:47:37.720 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2020-10-15T16:47:37.720 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:47:37.714+0000 7f44cb7c40c0 0 bucket checkpoint complete 2020-10-15T16:47:37.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:37.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:37.746 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:47:37.795 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:47:37.795 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:47:38.050 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:38.045+0000 7f14ba4660c0 1 bucket sync caught up with source: 2020-10-15T16:47:38.051 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000004.17.9, , , , , ] 2020-10-15T16:47:38.051 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2020-10-15T16:47:38.051 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:38.045+0000 7f14ba4660c0 0 bucket checkpoint complete 2020-10-15T16:47:38.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-19 zones={test-zone2, test-zone1} 2020-10-15T16:47:38.091 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:47:38.138 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:47:38.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:47:38.146 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:47:38.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-19 zones={test-zone2, test-zone1} 2020-10-15T16:47:38.346 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-19 zones={test-zone3, test-zone1} 2020-10-15T16:47:38.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:47:38.406 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:47:38.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:47:38.414 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:47:38.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-19 zones={test-zone3, test-zone1} 2020-10-15T16:47:38.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-19 zones={test-zone3, test-zone2} 2020-10-15T16:47:38.611 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:47:38.662 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:47:38.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:47:38.669 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:47:38.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:47:38.865 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-19 zones={test-zone3, test-zone2} 2020-10-15T16:47:40.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:40.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:47:40.332 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:47:40.383 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:47:40.383 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:47:40.630 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:40.625+0000 7fbe0a32f0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:40.630 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000010.25.11, , ] 2020-10-15T16:47:40.630 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000028.65.11, , ] 2020-10-15T16:47:45.634 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:45.629+0000 7fbe0a32f0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:45.635 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000010.25.11, , ] 2020-10-15T16:47:45.635 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000028.65.11, , ] 2020-10-15T16:47:50.638 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:50.633+0000 7fbe0a32f0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:50.638 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000010.25.11, , ] 2020-10-15T16:47:50.638 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000028.65.11, , ] 2020-10-15T16:47:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:47:55.637+0000 7fbe0a32f0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:47:55.642 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000010.25.11, , ] 2020-10-15T16:47:55.642 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000028.65.11, , ] 2020-10-15T16:48:00.646 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:00.641+0000 7fbe0a32f0c0 1 bucket sync caught up with source: 2020-10-15T16:48:00.646 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000043.105.5, , ] 2020-10-15T16:48:00.646 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000028.65.11, , ] 2020-10-15T16:48:00.647 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:00.641+0000 7fbe0a32f0c0 0 bucket checkpoint complete 2020-10-15T16:48:00.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:00.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:00.681 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:48:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:48:00.730 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:48:00.983 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:00.978+0000 7f7da57ba0c0 1 bucket sync caught up with source: 2020-10-15T16:48:00.983 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:00.983 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:00.983 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:00.978+0000 7f7da57ba0c0 0 bucket checkpoint complete 2020-10-15T16:48:01.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:01.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:01.009 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:48:01.057 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:48:01.057 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:48:01.324 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:01.317+0000 7fdb46b810c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:01.325 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000033.81.5, , ] 2020-10-15T16:48:01.325 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:06.328 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:06.321+0000 7fdb46b810c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:06.329 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000033.81.5, , ] 2020-10-15T16:48:06.329 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:11.333 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:11.329+0000 7fdb46b810c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:11.334 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000033.81.5, , ] 2020-10-15T16:48:11.334 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:16.338 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:16.333+0000 7fdb46b810c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:16.339 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000033.81.5, , ] 2020-10-15T16:48:16.339 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:21.341 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:21.337+0000 7fdb46b810c0 1 bucket sync caught up with source: 2020-10-15T16:48:21.342 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:21.342 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000047.115.9, , ] 2020-10-15T16:48:21.342 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:21.337+0000 7fdb46b810c0 0 bucket checkpoint complete 2020-10-15T16:48:21.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:21.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:21.559 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:48:21.605 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:48:21.606 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:48:21.853 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:21.849+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:21.853 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:21.854 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:26.857 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:26.853+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:26.857 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:26.858 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:31.860 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:31.857+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:31.861 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:31.861 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:36.865 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:36.861+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:36.865 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:36.865 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:41.869 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:41.865+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:41.869 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:41.869 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:46.873 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:46.869+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:46.873 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:46.874 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:51.877 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:51.873+0000 7f186fb8c0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:48:51.877 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000043.113.5, , ] 2020-10-15T16:48:51.878 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:56.881 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:56.876+0000 7f186fb8c0c0 1 bucket sync caught up with source: 2020-10-15T16:48:56.881 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:56.881 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:56.881 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:56.876+0000 7f186fb8c0c0 0 bucket checkpoint complete 2020-10-15T16:48:56.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:56.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:56.919 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:48:56.966 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:48:56.966 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:48:57.191 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:57.184+0000 7f118fb930c0 1 bucket sync caught up with source: 2020-10-15T16:48:57.191 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:57.191 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000044.115.9, , ] 2020-10-15T16:48:57.192 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:57.184+0000 7f118fb930c0 0 bucket checkpoint complete 2020-10-15T16:48:57.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:57.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:57.221 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:48:57.264 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:48:57.264 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:48:57.501 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:57.496+0000 7fb3ce5ed0c0 1 bucket sync caught up with source: 2020-10-15T16:48:57.501 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000045.120.9, , ] 2020-10-15T16:48:57.502 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000045.120.9, , ] 2020-10-15T16:48:57.502 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:48:57.496+0000 7fb3ce5ed0c0 0 bucket checkpoint complete 2020-10-15T16:48:57.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:57.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:48:57.528 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:48:57.576 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:48:57.577 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:48:57.824 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:57.821+0000 7f45118220c0 1 bucket sync caught up with source: 2020-10-15T16:48:57.825 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000004.17.9, , , 00000000045.120.9, , ] 2020-10-15T16:48:57.825 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000004.17.9, , , 00000000045.120.9, , ] 2020-10-15T16:48:57.825 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:48:57.821+0000 7f45118220c0 0 bucket checkpoint complete 2020-10-15T16:48:57.858 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-19 zones={test-zone2, test-zone1} 2020-10-15T16:48:57.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:48:57.882 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:48:57.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:48:57.902 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:48:57.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:48:57.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:48:57.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:48:58.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:48:58.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:58.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:48:59.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:00.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:01.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:02.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:03.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:04.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:05.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:06.793 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-19 zones={test-zone2, test-zone1} 2020-10-15T16:49:06.794 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-19 zones={test-zone3, test-zone1} 2020-10-15T16:49:06.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:49:06.811 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:49:06.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.820 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:49:06.828 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:49:06.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:06.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:49:06.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:49:06.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:49:06.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:07.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:08.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:09.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:10.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:11.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:12.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:13.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:14.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-19 zones={test-zone3, test-zone1} 2020-10-15T16:49:15.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-19 zones={test-zone3, test-zone2} 2020-10-15T16:49:15.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:49:15.498 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:49:15.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.508 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:49:15.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-15T16:49:15.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:15.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:49:15.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:49:15.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-15T16:49:15.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:15.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:16.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:17.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:18.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:19.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:20.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:21.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:22.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:23.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:24.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:24.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-19 zones={test-zone3, test-zone2} 2020-10-15T16:49:24.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:24.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:24.157 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:24.197 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:24.197 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:24.445 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:24.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:24.446 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:24.446 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:48:18.750337Z" 2020-10-15T16:49:24.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:24.447 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:24.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:24.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:24.448 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:24.448 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:49:24.448 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:24.448 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:24.449 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-15T16:49:24.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:24.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:24.451 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:24.494 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:24.495 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:24.761 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:24.762 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:24.762 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:24.762 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:24.762 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:24.762 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 2547530715017092207 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:24.763 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:48:18.750337Z" 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:24.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:24.765 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:24.766 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:49:24.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:24.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:24.767 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000" 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:24.768 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:49:24.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T16:49:24.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:24.771 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:24.772 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-15T16:49:24.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:24.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:24.773 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:24.812 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:24.813 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:25.046 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:25.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.047 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:25.047 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:48:18.750337Z" 2020-10-15T16:49:25.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.047 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.049 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:25.049 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:49:25.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.049 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:25.050 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-15T16:49:25.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:25.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:25.051 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:25.136 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:25.136 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:25.376 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:49:25.377 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:49:25.377 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:49:25.377 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:49:25.377 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:49:25.377 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 1541412607139480694 2020-10-15T16:49:25.378 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.378 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:49:25.378 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.378 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:49:25.378 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:25.378 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:48:18.750337Z" 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:25.380 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:25.381 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:25.382 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:25.382 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:25.382 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:25.382 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:25.382 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000" 2020-10-15T16:49:25.382 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:25.383 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.384 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:49:25.385 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:49:25.385 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:49:25.385 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.385 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:49:25.386 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-15T16:49:25.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:25.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:25.388 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:25.467 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:25.467 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:25.709 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:48:18.750337Z" 2020-10-15T16:49:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:25.712 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:25.712 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:49:25.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:25.712 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:25.713 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-15T16:49:25.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:25.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:25.714 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:25.758 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:25.758 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:26.136 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:26.136 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 16587652486901249284 2020-10-15T16:49:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:26.138 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:48:18.750337Z" 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:26.139 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:26.141 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000" 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:26.142 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:49:26.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.144 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:26.145 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-15T16:49:26.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:26.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:26.147 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:26.229 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:26.230 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:26.489 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:49:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-15T16:49:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:48:40.671004Z" 2020-10-15T16:49:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:49:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "0.000000" 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:49:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:49:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:26.492 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:49:26.493 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-15T16:49:26.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:26.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:26.494 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:26.537 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:26.537 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:26.783 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:49:26.784 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:49:26.784 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:49:26.784 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:49:26.784 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:49:26.784 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 6408475988550126921 2020-10-15T16:49:26.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-15T16:49:26.785 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:48:40.671004Z" 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:26.786 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:26.787 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.788 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000" 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:26.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:26.790 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:49:26.791 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:49:26.792 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:26.792 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:49:26.793 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-15T16:49:26.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:26.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:26.794 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:26.833 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:26.833 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:27.097 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:49:27.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:27.098 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-15T16:49:27.098 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:48:40.671004Z" 2020-10-15T16:49:27.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:27.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:27.099 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:49:27.099 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:49:27.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:27.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:27.099 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:27.100 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "0.000000" 2020-10-15T16:49:27.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:27.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:27.100 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:49:27.101 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:49:27.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:27.101 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:49:27.103 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-15T16:49:27.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:27.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:27.105 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:27.150 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:27.150 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:27.406 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:49:27.407 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:49:27.407 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:49:27.407 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:49:27.407 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 5492724315337703495 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:27.408 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:48:40.671004Z" 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.410 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:49:27.410 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:27.410 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:27.410 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:49:27.410 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.411 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:27.412 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000" 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:27.413 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:49:27.414 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:27.414 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:27.414 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:27.414 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:49:27.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.415 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:49:27.416 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-15T16:49:27.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:27.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:27.418 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:27.497 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:27.498 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:27.748 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:27.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.749 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:27.749 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:48:18.685792Z" 2020-10-15T16:49:27.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:27.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:27.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:27.751 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:27.751 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:49:27.751 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:27.751 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:27.752 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-15T16:49:27.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:27.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:27.753 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:27.837 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:27.837 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:28.082 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:49:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:49:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:49:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:49:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:49:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 4508126650953487876 2020-10-15T16:49:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:49:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:49:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:48:18.685792Z" 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:49:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000" 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:49:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.092 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:49:28.093 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-15T16:49:28.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:28.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:28.094 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:28.173 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:28.174 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:28.409 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:28.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.410 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:28.410 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:48:18.685792Z" 2020-10-15T16:49:28.410 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:28.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:28.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:28.412 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:28.412 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:49:28.412 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:28.412 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:28.413 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-15T16:49:28.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:28.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:28.414 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:28.457 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:28.457 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:28.739 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:28.740 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:28.740 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:28.740 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:28.740 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:28.740 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 11637474988163193673 2020-10-15T16:49:28.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:28.741 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:28.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:28.741 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:28.741 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:28.742 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.742 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-10-15T16:49:28.742 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:28.742 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:28.743 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:28.743 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:48:18.685792Z" 2020-10-15T16:49:28.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:28.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:28.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:28.744 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:28.744 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:28.745 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.745 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:49:28.745 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:28.745 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:28.746 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:28.746 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:49:28.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:28.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:28.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:28.747 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:28.747 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:28.747 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.748 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-15T16:49:28.748 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:28.748 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:28.749 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:28.749 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000" 2020-10-15T16:49:28.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:28.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:28.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:28.750 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:28.750 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:28.750 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:49:28.751 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:49:28.751 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:28.751 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:28.752 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:28.752 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:49:28.752 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:28.752 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:28.753 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:28.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:28.753 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:28.753 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T16:49:28.753 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T16:49:28.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:28.754 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:28.755 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-15T16:49:28.757 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-10-15T16:49:28.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:28.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:28.789 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:28.824 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:28.824 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:29.012 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:49:29.044 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780568.783569_164.1", 2020-10-15T16:49:29.045 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:49:28.783569Z" 2020-10-15T16:49:29.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.046 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:29.046 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:29.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.047 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:29.047 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:29.047 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.047 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:29.048 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:49:29.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:29.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:29.050 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:29.091 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:29.341 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:49:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780568.783569_164.1", 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.783569Z", 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:29.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:29.347 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:29.349 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:29.350 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:49:29.351 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:49:29.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:29.351 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:29.352 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:49:29.353 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:49:29.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:29.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:29.354 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:29.394 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:29.395 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:29.642 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:49:29.642 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:49:29.643 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:49:29.643 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:49:29.643 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:49:29.643 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:49:29.643 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:29.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:29.644 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780568.783569_164.1", 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:29.646 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:29.647 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.783569Z", 2020-10-15T16:49:29.647 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:29.647 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.648 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:49:29.648 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:29.648 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:29.649 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:29.650 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:49:29.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:29.652 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:49:29.652 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:49:29.652 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:49:29.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:29.652 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:49:29.653 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:49:29.653 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:49:29.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:29.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:29.655 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:29.694 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:30.016 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:30.065 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:30.065 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:30.065 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:30.065 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:30.066 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:30.067 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:30.068 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780568.783569_164.1", 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.783569Z", 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:30.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:30.070 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:30.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:30.072 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:30.073 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:49:30.074 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:49:30.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:30.074 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:30.075 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:49:30.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:30.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:30.127 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:30.407 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:30.404+0000 7f2987a400c0 1 bucket sync caught up with source: 2020-10-15T16:49:30.408 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:30.408 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:30.408 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:30.404+0000 7f2987a400c0 0 bucket checkpoint complete 2020-10-15T16:49:30.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:30.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:30.441 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:30.491 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:30.491 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:30.699 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:30.691+0000 7f06d86230c0 1 bucket sync caught up with source: 2020-10-15T16:49:30.700 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:30.700 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:30.700 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:30.691+0000 7f06d86230c0 0 bucket checkpoint complete 2020-10-15T16:49:30.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:30.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:30.727 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:30.776 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:30.776 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:31.028+0000 7f699fbae0c0 1 bucket sync caught up with source: 2020-10-15T16:49:31.031 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:31.032 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:31.028+0000 7f699fbae0c0 0 bucket checkpoint complete 2020-10-15T16:49:31.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:31.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:31.143 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:31.188 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:31.189 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:31.429 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:31.423+0000 7f11fec9a0c0 1 bucket sync caught up with source: 2020-10-15T16:49:31.430 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:49:31.430 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:49:31.430 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:31.423+0000 7f11fec9a0c0 0 bucket checkpoint complete 2020-10-15T16:49:31.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:31.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:31.460 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:31.504 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:31.504 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:31.744 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:31.735+0000 7f81738360c0 1 bucket sync caught up with source: 2020-10-15T16:49:31.744 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:49:31.744 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:49:31.744 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:31.735+0000 7f81738360c0 0 bucket checkpoint complete 2020-10-15T16:49:31.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:31.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:31.773 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:31.818 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:31.818 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:32.036 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:32.027+0000 7f3d548470c0 1 bucket sync caught up with source: 2020-10-15T16:49:32.036 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-10-15T16:49:32.036 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.18.5, , ] 2020-10-15T16:49:32.036 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:32.027+0000 7f3d548470c0 0 bucket checkpoint complete 2020-10-15T16:49:32.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:32.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:32.066 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:32.116 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:32.116 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:32.346 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:32.344+0000 7f73f7ef20c0 1 bucket sync caught up with source: 2020-10-15T16:49:32.347 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.18.5, , ] 2020-10-15T16:49:32.347 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.18.5, , ] 2020-10-15T16:49:32.347 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:32.344+0000 7f73f7ef20c0 0 bucket checkpoint complete 2020-10-15T16:49:32.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone2, test-zone1} 2020-10-15T16:49:32.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:49:32.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:32.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:49:32.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:32.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:32.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:32.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:32.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone2, test-zone1} 2020-10-15T16:49:32.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone3, test-zone1} 2020-10-15T16:49:32.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:49:32.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:32.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:49:32.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:32.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:32.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:32.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:32.880 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone3, test-zone1} 2020-10-15T16:49:32.880 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone3, test-zone2} 2020-10-15T16:49:32.892 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:49:32.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:32.939 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:49:32.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:49:33.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:33.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:33.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:49:33.132 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone3, test-zone2} 2020-10-15T16:49:33.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:33.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:33.147 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:33.183 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:33.183 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:33.405 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:49:33.432 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:49:33.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:33.433 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:33.433 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:49:33.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:33.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:33.434 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780573.141127_169.1", 2020-10-15T16:49:33.434 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:49:33.141127Z" 2020-10-15T16:49:33.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:33.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:33.434 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:33.434 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:33.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:33.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:33.435 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:33.435 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:49:33.435 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:33.435 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:49:33.436 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:49:33.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:33.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:33.437 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:33.477 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:33.477 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:33.750 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:33.750 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:33.751 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:33.751 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:33.751 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:33.751 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:49:33.751 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:33.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:33.752 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:33.753 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780573.141127_169.1", 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:33.754 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:33.141127Z", 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:33.755 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:33.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:33.757 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:33.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:33.759 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:33.759 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:49:33.759 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:49:33.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:33.759 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:33.760 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:49:33.761 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:49:33.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:33.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:33.762 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:33.802 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:33.802 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:34.040 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:49:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:49:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:49:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:49:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:49:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:49:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:49:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:49:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:49:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:49:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:49:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780573.141127_169.1", 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:33.141127Z", 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:49:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:49:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:49:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:49:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:49:34.050 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:49:34.051 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:49:34.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:49:34.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:34.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:49:34.053 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:49:34.094 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:34.094 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:34.357 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:49:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:49:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:49:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:49:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:49:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:49:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780573.141127_169.1", 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:33.141127Z", 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:49:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:49:34.366 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:49:34.367 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:49:34.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:34.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:34.419 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:34.465 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:34.466 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:34.726 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:34.724+0000 7f41e2c4b0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:49:34.726 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:34.727 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:39.730 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:39.728+0000 7f41e2c4b0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:49:39.731 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:39.731 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:44.734 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:44.732+0000 7f41e2c4b0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:49:44.735 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:44.735 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:49.738 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:49.736+0000 7f41e2c4b0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:49:49.739 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2020-10-15T16:49:49.739 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:54.743 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:54.740+0000 7f41e2c4b0c0 1 bucket sync caught up with source: 2020-10-15T16:49:54.743 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:54.744 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:54.744 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:54.740+0000 7f41e2c4b0c0 0 bucket checkpoint complete 2020-10-15T16:49:54.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:54.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:54.781 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:55.046 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:55.043+0000 7f62c10320c0 1 bucket sync caught up with source: 2020-10-15T16:49:55.046 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:55.046 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:55.046 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:55.043+0000 7f62c10320c0 0 bucket checkpoint complete 2020-10-15T16:49:55.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:55.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:55.072 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:55.120 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:55.121 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:55.372 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:55.372+0000 7f67a9cb10c0 1 bucket sync caught up with source: 2020-10-15T16:49:55.373 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:55.373 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2020-10-15T16:49:55.373 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:49:55.372+0000 7f67a9cb10c0 0 bucket checkpoint complete 2020-10-15T16:49:55.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:55.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:49:55.499 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:49:55.546 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:49:55.547 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:49:55.784 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:49:55.779+0000 7f027f0320c0 1 waiting for incremental sync to catch up: 2020-10-15T16:49:55.784 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:49:55.784 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:00.787 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:00.783+0000 7f027f0320c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:00.788 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:50:00.788 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:05.791 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:05.786+0000 7f027f0320c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:05.792 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2020-10-15T16:50:05.792 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:10.795 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:10.790+0000 7f027f0320c0 1 bucket sync caught up with source: 2020-10-15T16:50:10.796 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:10.796 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:10.796 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:10.790+0000 7f027f0320c0 0 bucket checkpoint complete 2020-10-15T16:50:10.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:10.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:10.827 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:10.872 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:10.872 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:11.093 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:11.090+0000 7fa6657760c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:11.093 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:11.094 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:16.096 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:16.090+0000 7fa6657760c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:16.097 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:16.097 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:21.101 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:21.094+0000 7fa6657760c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:21.101 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:21.101 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:26.104 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:26.098+0000 7fa6657760c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:26.104 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:26.105 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:31.108 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:31.102+0000 7fa6657760c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:31.109 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2020-10-15T16:50:31.109 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:36.112 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:36.106+0000 7fa6657760c0 1 bucket sync caught up with source: 2020-10-15T16:50:36.112 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:36.113 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:36.113 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:36.106+0000 7fa6657760c0 0 bucket checkpoint complete 2020-10-15T16:50:36.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:36.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:36.143 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:36.230 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:36.230 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:36.468 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:36.461+0000 7f8cdef0c0c0 1 bucket sync caught up with source: 2020-10-15T16:50:36.469 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-10-15T16:50:36.469 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000004.24.9, , ] 2020-10-15T16:50:36.469 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:36.461+0000 7f8cdef0c0c0 0 bucket checkpoint complete 2020-10-15T16:50:36.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:36.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:36.496 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:36.544 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:36.544 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:36.816 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:50:36.811+0000 7f5622efa0c0 1 bucket sync caught up with source: 2020-10-15T16:50:36.816 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000004.24.9, , ] 2020-10-15T16:50:36.817 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000004.24.9, , ] 2020-10-15T16:50:36.817 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:50:36.811+0000 7f5622efa0c0 0 bucket checkpoint complete 2020-10-15T16:50:36.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone2, test-zone1} 2020-10-15T16:50:36.860 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:50:36.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:36.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:36.912 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:50:36.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:36.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:36.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:36.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.253 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone2, test-zone1} 2020-10-15T16:50:37.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone3, test-zone1} 2020-10-15T16:50:37.263 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:50:37.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:50:37.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.685 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone3, test-zone1} 2020-10-15T16:50:37.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone3, test-zone2} 2020-10-15T16:50:37.696 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:50:37.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.743 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:50:37.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:50:37.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:37.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:38.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:50:38.076 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone3, test-zone2} 2020-10-15T16:50:38.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:38.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:38.091 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:50:38.128 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:38.128 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:38.342 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:50:38.382 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:50:38.382 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:38.382 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:50:38.383 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:50:38.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:38.383 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:38.384 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780638.085516_186.1", 2020-10-15T16:50:38.384 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:50:38.085516Z" 2020-10-15T16:50:38.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:38.384 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:38.385 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:50:38.385 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:50:38.385 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:38.385 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:38.386 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:50:38.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:50:38.386 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:38.387 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:50:38.388 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:50:38.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:38.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:38.390 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:50:38.432 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:38.433 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:38.714 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:50:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:50:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:50:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:50:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:50:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:50:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:50:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:50:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:50:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780638.085516_186.1", 2020-10-15T16:50:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:50:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:50:38.085516Z", 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:50:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:50:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:50:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:50:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:50:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:38.723 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:50:38.724 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:50:38.725 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:50:38.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:38.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:38.726 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:50:38.765 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:38.766 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:39.021 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:50:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:50:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:50:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:50:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:50:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:50:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:50:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:50:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:50:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:50:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:50:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:50:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:50:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:50:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:50:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:50:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:50:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:50:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:50:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:50:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780638.085516_186.1", 2020-10-15T16:50:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:50:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:50:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:50:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:50:38.085516Z", 2020-10-15T16:50:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:50:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:50:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:50:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:50:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:50:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:50:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:50:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:50:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:50:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:50:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:50:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:50:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:50:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:50:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:50:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:50:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:50:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:50:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:50:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:50:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:50:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:50:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:50:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:50:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:50:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:50:39.039 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:50:39.040 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:50:39.041 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:50:39.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:39.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:50:39.043 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:50:39.085 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:39.085 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:39.360 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:50:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:50:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:50:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:50:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:50:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:50:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:50:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780638.085516_186.1", 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:50:38.085516Z", 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:50:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:50:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:50:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:50:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:50:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:50:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:50:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:50:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:50:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:50:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:50:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:50:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:50:39.372 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:50:39.373 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:50:39.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:39.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:39.423 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:39.472 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:39.472 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:39.713 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:50:39.711+0000 7f2b8c2730c0 1 bucket sync caught up with source: 2020-10-15T16:50:39.713 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000005.16.17, , ] 2020-10-15T16:50:39.714 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2020-10-15T16:50:39.714 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:50:39.711+0000 7f2b8c2730c0 0 bucket checkpoint complete 2020-10-15T16:50:39.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:39.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:39.745 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:39.793 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:39.794 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:40.033 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:40.025+0000 7fdb02c810c0 1 bucket sync caught up with source: 2020-10-15T16:50:40.033 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000005.16.17, , ] 2020-10-15T16:50:40.033 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2020-10-15T16:50:40.033 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:40.029+0000 7fdb02c810c0 0 bucket checkpoint complete 2020-10-15T16:50:40.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:40.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:40.062 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:40.109 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:40.109 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:40.362 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:50:40.359+0000 7f703bf520c0 1 bucket sync caught up with source: 2020-10-15T16:50:40.362 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000005.16.17, , ] 2020-10-15T16:50:40.362 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2020-10-15T16:50:40.362 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:50:40.359+0000 7f703bf520c0 0 bucket checkpoint complete 2020-10-15T16:50:40.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:40.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:40.477 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:40.525 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:40.525 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:40.777 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:40.773+0000 7fa4ccdd00c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:40.778 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:40.778 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000006.22.17, , ] 2020-10-15T16:50:45.781 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:45.777+0000 7fa4ccdd00c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:45.782 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:45.782 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000006.22.17, , ] 2020-10-15T16:50:50.785 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:50.781+0000 7fa4ccdd00c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:50.785 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000004.17.9, , ] 2020-10-15T16:50:50.786 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000006.22.17, , ] 2020-10-15T16:50:55.789 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:55.785+0000 7fa4ccdd00c0 1 bucket sync caught up with source: 2020-10-15T16:50:55.789 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000007.24.14, , ] 2020-10-15T16:50:55.789 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000006.22.17, , ] 2020-10-15T16:50:55.789 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:55.785+0000 7fa4ccdd00c0 0 bucket checkpoint complete 2020-10-15T16:50:55.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:55.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:55.821 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:55.865 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:55.865 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:56.077 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:56.073+0000 7f5e6b3c20c0 1 bucket sync caught up with source: 2020-10-15T16:50:56.078 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000007.24.14, , ] 2020-10-15T16:50:56.078 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000007.24.14, , ] 2020-10-15T16:50:56.078 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:56.073+0000 7f5e6b3c20c0 0 bucket checkpoint complete 2020-10-15T16:50:56.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:56.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:50:56.106 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:50:56.150 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:50:56.151 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:50:56.368 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:50:56.361+0000 7fbabcaec0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:50:56.368 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000006.29.17, , ] 2020-10-15T16:50:56.368 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:01.372 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:01.365+0000 7fbabcaec0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:51:01.372 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000006.29.17, , ] 2020-10-15T16:51:01.373 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:06.376 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:06.369+0000 7fbabcaec0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:51:06.377 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000006.29.17, , ] 2020-10-15T16:51:06.377 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:11.380 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:11.373+0000 7fbabcaec0c0 1 waiting for incremental sync to catch up: 2020-10-15T16:51:11.381 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000006.29.17, , ] 2020-10-15T16:51:11.381 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:16.384 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:16.376+0000 7fbabcaec0c0 1 bucket sync caught up with source: 2020-10-15T16:51:16.384 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:16.385 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:16.385 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:16.376+0000 7fbabcaec0c0 0 bucket checkpoint complete 2020-10-15T16:51:16.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:16.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:16.416 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:16.465 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:16.465 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:16.712 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:16.706+0000 7f040b1ac0c0 1 bucket sync caught up with source: 2020-10-15T16:51:16.713 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:16.713 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000007.31.14, , ] 2020-10-15T16:51:16.713 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:16.706+0000 7f040b1ac0c0 0 bucket checkpoint complete 2020-10-15T16:51:16.747 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone2, test-zone1} 2020-10-15T16:51:16.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:16.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:16.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:16.808 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:16.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:16.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:16.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:16.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:16.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone2, test-zone1} 2020-10-15T16:51:17.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone3, test-zone1} 2020-10-15T16:51:17.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:17.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:17.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.528 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone3, test-zone1} 2020-10-15T16:51:17.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-20 zones={test-zone3, test-zone2} 2020-10-15T16:51:17.539 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:17.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.587 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:17.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:17.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:17.933 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-20 zones={test-zone3, test-zone2} 2020-10-15T16:51:17.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-10-15T16:51:17.935 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-21 2020-10-15T16:51:17.969 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-22 2020-10-15T16:51:18.029 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-23 2020-10-15T16:51:18.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:18.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:18.124 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:18.174 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:18.174 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:18.378 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:51:18.411 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:51:18.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:18.412 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:18.412 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:51:18.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:18.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:18.412 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780678.113514_207.1", 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:18.113514Z" 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:18.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:18.414 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:18.414 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:18.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:18.415 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:51:18.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:51:18.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:18.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:18.417 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:18.725 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780678.113514_207.1", 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.113514Z", 2020-10-15T16:51:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:18.735 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:18.736 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:51:18.737 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:51:18.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:18.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:18.738 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:18.778 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:18.778 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:19.015 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:51:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:51:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:51:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:51:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:51:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:51:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:51:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:51:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:51:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780678.113514_207.1", 2020-10-15T16:51:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.113514Z", 2020-10-15T16:51:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:51:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:51:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:51:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:51:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:51:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:51:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:19.030 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:51:19.031 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:51:19.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:51:19.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:19.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:19.034 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:19.346 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:19.347 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:19.347 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:19.347 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:19.347 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:19.347 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:19.348 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:19.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:19.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:19.350 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:19.350 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:19.350 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:19.350 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780678.113514_207.1", 2020-10-15T16:51:19.350 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.113514Z", 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:19.351 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:19.352 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:19.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:19.355 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:19.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:51:19.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:19.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:19.575 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:19.628 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:19.629 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:19.841 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:19.838+0000 7f72464d20c0 1 bucket sync caught up with source: 2020-10-15T16:51:19.841 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.12.11, , ] 2020-10-15T16:51:19.841 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.11, , ] 2020-10-15T16:51:19.841 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:19.838+0000 7f72464d20c0 0 bucket checkpoint complete 2020-10-15T16:51:19.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:19.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:19.864 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:19.912 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:19.913 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:20.126 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:20.120+0000 7f6a3faa50c0 1 bucket sync caught up with source: 2020-10-15T16:51:20.127 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.12.11, , ] 2020-10-15T16:51:20.127 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.12.11, , ] 2020-10-15T16:51:20.127 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:20.120+0000 7f6a3faa50c0 0 bucket checkpoint complete 2020-10-15T16:51:20.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:20.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:20.153 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:20.200 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:20.201 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:20.461 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:20.458+0000 7fc3b60040c0 1 bucket sync caught up with source: 2020-10-15T16:51:20.462 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.12.11, , ] 2020-10-15T16:51:20.462 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.12.11, , ] 2020-10-15T16:51:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:20.458+0000 7fc3b60040c0 0 bucket checkpoint complete 2020-10-15T16:51:20.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:20.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:20.562 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:20.610 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:20.611 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:20.833 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:20.828+0000 7f7d1f9fa0c0 1 bucket sync caught up with source: 2020-10-15T16:51:20.834 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-10-15T16:51:20.834 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-10-15T16:51:20.834 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:20.828+0000 7f7d1f9fa0c0 0 bucket checkpoint complete 2020-10-15T16:51:20.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:20.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:20.865 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:20.910 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:20.911 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:21.140 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:21.132+0000 7fa1bb2360c0 1 bucket sync caught up with source: 2020-10-15T16:51:21.141 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-10-15T16:51:21.141 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-10-15T16:51:21.141 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:21.132+0000 7fa1bb2360c0 0 bucket checkpoint complete 2020-10-15T16:51:21.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:21.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:21.170 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:21.258 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:21.259 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:21.500 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:21.492+0000 7f567a86a0c0 1 bucket sync caught up with source: 2020-10-15T16:51:21.501 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:21.501 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:21.501 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:21.492+0000 7f567a86a0c0 0 bucket checkpoint complete 2020-10-15T16:51:21.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:21.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:21.527 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:21.578 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:21.579 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:21.834 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:21.830+0000 7f73edccc0c0 1 bucket sync caught up with source: 2020-10-15T16:51:21.834 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:21.835 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:21.835 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:21.830+0000 7f73edccc0c0 0 bucket checkpoint complete 2020-10-15T16:51:21.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-21 zones={test-zone2, test-zone1} 2020-10-15T16:51:21.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:21.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:21.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:21.928 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:21.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:21.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:21.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:21.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.076 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-21 zones={test-zone2, test-zone1} 2020-10-15T16:51:22.076 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-21 zones={test-zone3, test-zone1} 2020-10-15T16:51:22.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:22.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:22.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.279 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-21 zones={test-zone3, test-zone1} 2020-10-15T16:51:22.279 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-21 zones={test-zone3, test-zone2} 2020-10-15T16:51:22.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:22.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:22.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:22.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:22.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-21 zones={test-zone3, test-zone2} 2020-10-15T16:51:22.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:22.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:22.495 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:22.541 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:22.541 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:22.800 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:22.794+0000 7f4b9638b0c0 1 bucket sync caught up with source: 2020-10-15T16:51:22.800 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2020-10-15T16:51:22.800 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2020-10-15T16:51:22.800 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:22.794+0000 7f4b9638b0c0 0 bucket checkpoint complete 2020-10-15T16:51:22.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:22.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:22.833 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:22.880 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:22.880 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:23.109 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:23.104+0000 7f75c2c910c0 1 bucket sync caught up with source: 2020-10-15T16:51:23.109 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2020-10-15T16:51:23.110 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2020-10-15T16:51:23.110 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:23.104+0000 7f75c2c910c0 0 bucket checkpoint complete 2020-10-15T16:51:23.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:23.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:23.137 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:23.184 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:23.185 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:23.417 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:23.414+0000 7f8576cd10c0 1 bucket sync caught up with source: 2020-10-15T16:51:23.418 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2020-10-15T16:51:23.418 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2020-10-15T16:51:23.418 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:23.414+0000 7f8576cd10c0 0 bucket checkpoint complete 2020-10-15T16:51:23.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:23.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:23.550 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:23.598 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:23.598 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:23.828 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:23.820+0000 7f45107c20c0 1 bucket sync caught up with source: 2020-10-15T16:51:23.828 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.28.11, , ] 2020-10-15T16:51:23.829 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.28.11, , ] 2020-10-15T16:51:23.829 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:23.820+0000 7f45107c20c0 0 bucket checkpoint complete 2020-10-15T16:51:23.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:23.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:23.861 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:23.907 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:23.907 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:24.130 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:24.124+0000 7f8c725440c0 1 bucket sync caught up with source: 2020-10-15T16:51:24.130 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.28.11, , ] 2020-10-15T16:51:24.130 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.28.11, , ] 2020-10-15T16:51:24.130 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:24.124+0000 7f8c725440c0 0 bucket checkpoint complete 2020-10-15T16:51:24.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:24.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:24.160 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:24.202 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:24.202 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:24.447 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:24.440+0000 7fcd1f5ce0c0 1 bucket sync caught up with source: 2020-10-15T16:51:24.447 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-10-15T16:51:24.447 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-10-15T16:51:24.447 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:24.440+0000 7fcd1f5ce0c0 0 bucket checkpoint complete 2020-10-15T16:51:24.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:24.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:24.473 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:24.522 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:24.523 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:24.762 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:24.758+0000 7fbc327070c0 1 bucket sync caught up with source: 2020-10-15T16:51:24.763 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2020-10-15T16:51:24.763 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2020-10-15T16:51:24.763 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:24.758+0000 7fbc327070c0 0 bucket checkpoint complete 2020-10-15T16:51:24.794 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-22 zones={test-zone2, test-zone1} 2020-10-15T16:51:24.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:24.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:24.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:24.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:24.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:24.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:24.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:24.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:24.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-22 zones={test-zone2, test-zone1} 2020-10-15T16:51:25.005 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-22 zones={test-zone3, test-zone1} 2020-10-15T16:51:25.013 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:25.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.063 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:25.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-22 zones={test-zone3, test-zone1} 2020-10-15T16:51:25.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-22 zones={test-zone3, test-zone2} 2020-10-15T16:51:25.214 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:25.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:25.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:25.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:25.411 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-22 zones={test-zone3, test-zone2} 2020-10-15T16:51:25.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:25.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:25.413 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:25.721 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:25.718+0000 7f53ac53f0c0 1 bucket sync caught up with source: 2020-10-15T16:51:25.721 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2020-10-15T16:51:25.721 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2020-10-15T16:51:25.721 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:25.718+0000 7f53ac53f0c0 0 bucket checkpoint complete 2020-10-15T16:51:25.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:25.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:25.753 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:25.800 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:25.800 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:26.039 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:26.032+0000 7f32060900c0 1 bucket sync caught up with source: 2020-10-15T16:51:26.039 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2020-10-15T16:51:26.040 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2020-10-15T16:51:26.040 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:26.032+0000 7f32060900c0 0 bucket checkpoint complete 2020-10-15T16:51:26.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:26.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:26.059 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:26.106 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:26.107 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:26.360 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:26.354+0000 7f2d142f40c0 1 bucket sync caught up with source: 2020-10-15T16:51:26.360 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2020-10-15T16:51:26.360 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2020-10-15T16:51:26.360 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:26.354+0000 7f2d142f40c0 0 bucket checkpoint complete 2020-10-15T16:51:26.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:26.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:26.586 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:26.634 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:26.634 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:26.867 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:26.860+0000 7f1e67ee70c0 1 bucket sync caught up with source: 2020-10-15T16:51:26.867 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:26.867 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:26.868 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:26.860+0000 7f1e67ee70c0 0 bucket checkpoint complete 2020-10-15T16:51:26.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:26.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:26.900 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:26.944 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:26.945 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:27.164 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:27.156+0000 7f47f2cf10c0 1 bucket sync caught up with source: 2020-10-15T16:51:27.164 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:27.165 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2020-10-15T16:51:27.165 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:27.156+0000 7f47f2cf10c0 0 bucket checkpoint complete 2020-10-15T16:51:27.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:27.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:27.196 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:27.287 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:27.288 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:27.528 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:27.520+0000 7f594ab1c0c0 1 bucket sync caught up with source: 2020-10-15T16:51:27.528 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2020-10-15T16:51:27.528 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2020-10-15T16:51:27.528 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:27.520+0000 7f594ab1c0c0 0 bucket checkpoint complete 2020-10-15T16:51:27.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:27.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:27.559 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:27.854+0000 7f0663be80c0 1 bucket sync caught up with source: 2020-10-15T16:51:27.860 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2020-10-15T16:51:27.860 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2020-10-15T16:51:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:27.854+0000 7f0663be80c0 0 bucket checkpoint complete 2020-10-15T16:51:27.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-23 zones={test-zone2, test-zone1} 2020-10-15T16:51:27.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:27.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:27.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:27.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:27.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:27.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.100 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-23 zones={test-zone2, test-zone1} 2020-10-15T16:51:28.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-23 zones={test-zone3, test-zone1} 2020-10-15T16:51:28.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:28.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:28.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-23 zones={test-zone3, test-zone1} 2020-10-15T16:51:28.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-23 zones={test-zone3, test-zone2} 2020-10-15T16:51:28.304 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:28.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:28.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:28.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:28.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-23 zones={test-zone3, test-zone2} 2020-10-15T16:51:28.506 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-10-15T16:51:28.507 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-24 2020-10-15T16:51:28.538 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-25 2020-10-15T16:51:28.591 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-26 2020-10-15T16:51:28.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:28.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:28.758 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:28.809 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:28.810 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:28.994 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:51:29.015 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:51:29.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.016 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:29.016 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:51:29.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780688.747532_238.1", 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:28.747532Z" 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:29.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.018 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:29.018 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:29.018 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.018 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:51:29.019 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:51:29.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:29.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:29.021 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:29.058 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:29.059 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:29.327 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:29.327 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:29.327 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:29.327 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:29.328 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:29.329 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:29.330 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:29.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:29.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:29.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:29.330 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:29.330 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780688.747532_238.1", 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.747532Z", 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:29.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:29.332 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:29.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:29.334 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:29.335 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:29.335 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:29.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:29.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:29.335 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:29.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:29.336 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:29.337 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:51:29.338 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:51:29.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:29.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:29.339 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:29.378 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:29.379 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:29.622 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:51:29.623 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:51:29.623 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:51:29.623 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:51:29.623 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:51:29.623 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:29.623 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:29.624 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:29.625 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:29.625 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:51:29.625 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:29.625 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:29.625 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:29.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780688.747532_238.1", 2020-10-15T16:51:29.626 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.747532Z", 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.627 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:29.628 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:29.629 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:29.631 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:51:29.632 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:51:29.633 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:51:29.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:29.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:29.634 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:29.674 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:29.674 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:30.146 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:30.146 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:30.146 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:30.146 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:30.147 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:30.148 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:30.149 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:30.150 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780688.747532_238.1", 2020-10-15T16:51:30.150 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:30.150 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:30.150 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:30.150 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.747532Z", 2020-10-15T16:51:30.150 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:30.151 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:30.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:30.153 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:30.154 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:30.155 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:30.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:30.155 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:30.156 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:51:30.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:30.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:30.395 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:30.444 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:30.444 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:30.638+0000 7f905ae9c0c0 1 bucket sync caught up with source: 2020-10-15T16:51:30.642 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.17.6, , ] 2020-10-15T16:51:30.642 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.17.6, , ] 2020-10-15T16:51:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:30.638+0000 7f905ae9c0c0 0 bucket checkpoint complete 2020-10-15T16:51:30.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:30.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:30.668 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:30.715 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:30.715 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:30.913 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:30.904+0000 7fcfb894c0c0 1 bucket sync caught up with source: 2020-10-15T16:51:30.913 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.17.6, , ] 2020-10-15T16:51:30.913 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.17.6, , ] 2020-10-15T16:51:30.914 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:30.904+0000 7fcfb894c0c0 0 bucket checkpoint complete 2020-10-15T16:51:30.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:30.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:30.940 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:30.990 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:31.246 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:31.242+0000 7ff5e8df20c0 1 bucket sync caught up with source: 2020-10-15T16:51:31.246 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.17.6, , ] 2020-10-15T16:51:31.247 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.17.6, , ] 2020-10-15T16:51:31.247 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:31.242+0000 7ff5e8df20c0 0 bucket checkpoint complete 2020-10-15T16:51:31.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:31.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:31.367 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:31.413 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:31.414 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:31.642 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:31.636+0000 7fcaf3b230c0 1 bucket sync caught up with source: 2020-10-15T16:51:31.643 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.32.12, , ] 2020-10-15T16:51:31.643 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.32.12, , ] 2020-10-15T16:51:31.643 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:31.636+0000 7fcaf3b230c0 0 bucket checkpoint complete 2020-10-15T16:51:31.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:31.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:31.676 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:31.719 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:31.720 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:31.950 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:31.944+0000 7f7f447c60c0 1 bucket sync caught up with source: 2020-10-15T16:51:31.950 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.32.12, , ] 2020-10-15T16:51:31.950 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.32.12, , ] 2020-10-15T16:51:31.951 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:31.944+0000 7f7f447c60c0 0 bucket checkpoint complete 2020-10-15T16:51:31.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:31.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:31.983 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:32.069 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:32.069 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:32.284 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:32.276+0000 7f45a92a50c0 1 bucket sync caught up with source: 2020-10-15T16:51:32.284 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.25.12, , ] 2020-10-15T16:51:32.284 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.25.12, , ] 2020-10-15T16:51:32.285 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:32.276+0000 7f45a92a50c0 0 bucket checkpoint complete 2020-10-15T16:51:32.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:32.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:32.313 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:32.362 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:32.363 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:32.613 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:32.606+0000 7f8015e320c0 1 bucket sync caught up with source: 2020-10-15T16:51:32.838 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.25.12, , ] 2020-10-15T16:51:32.839 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.25.12, , ] 2020-10-15T16:51:32.839 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:32.606+0000 7f8015e320c0 0 bucket checkpoint complete 2020-10-15T16:51:32.840 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-24 zones={test-zone2, test-zone1} 2020-10-15T16:51:32.852 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:32.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:32.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:32.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:32.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:32.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-24 zones={test-zone2, test-zone1} 2020-10-15T16:51:32.971 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-24 zones={test-zone3, test-zone1} 2020-10-15T16:51:32.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:33.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:33.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:33.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:33.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:33.089 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-24 zones={test-zone3, test-zone1} 2020-10-15T16:51:33.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-24 zones={test-zone3, test-zone2} 2020-10-15T16:51:33.101 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:33.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:33.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:33.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:33.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:33.216 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-24 zones={test-zone3, test-zone2} 2020-10-15T16:51:33.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:33.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:33.218 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:33.538 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:33.534+0000 7fe481f200c0 1 bucket sync caught up with source: 2020-10-15T16:51:33.538 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-10-15T16:51:33.539 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-10-15T16:51:33.539 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:33.534+0000 7fe481f200c0 0 bucket checkpoint complete 2020-10-15T16:51:33.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:33.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:33.573 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:33.620 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:33.620 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:33.847 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:33.840+0000 7fbcddf200c0 1 bucket sync caught up with source: 2020-10-15T16:51:33.847 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-10-15T16:51:33.847 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-10-15T16:51:33.848 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:33.840+0000 7fbcddf200c0 0 bucket checkpoint complete 2020-10-15T16:51:33.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:33.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:33.876 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:33.927 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:33.928 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:34.188 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:34.186+0000 7fcce98e80c0 1 bucket sync caught up with source: 2020-10-15T16:51:34.189 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-10-15T16:51:34.189 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-10-15T16:51:34.189 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:34.186+0000 7fcce98e80c0 0 bucket checkpoint complete 2020-10-15T16:51:34.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:34.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:34.309 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:34.357 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:34.358 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:34.598 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:34.592+0000 7fba490db0c0 1 bucket sync caught up with source: 2020-10-15T16:51:34.599 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.30.6, , ] 2020-10-15T16:51:34.599 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.30.6, , ] 2020-10-15T16:51:34.599 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:34.592+0000 7fba490db0c0 0 bucket checkpoint complete 2020-10-15T16:51:34.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:34.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:34.630 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:34.675 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:34.675 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:34.910 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:34.904+0000 7fa988b6a0c0 1 bucket sync caught up with source: 2020-10-15T16:51:34.910 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.30.6, , ] 2020-10-15T16:51:34.910 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.30.6, , ] 2020-10-15T16:51:34.911 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:34.904+0000 7fa988b6a0c0 0 bucket checkpoint complete 2020-10-15T16:51:34.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:34.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:34.940 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:35.033 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:35.033 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:35.236 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:35.228+0000 7feccc2df0c0 1 bucket sync caught up with source: 2020-10-15T16:51:35.236 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.20.12, , ] 2020-10-15T16:51:35.237 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.20.12, , ] 2020-10-15T16:51:35.237 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:35.228+0000 7feccc2df0c0 0 bucket checkpoint complete 2020-10-15T16:51:35.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:35.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:35.265 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:35.314 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:35.314 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:35.568 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:35.566+0000 7f540601d0c0 1 bucket sync caught up with source: 2020-10-15T16:51:35.568 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.20.12, , ] 2020-10-15T16:51:35.569 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.20.12, , ] 2020-10-15T16:51:35.569 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:35.566+0000 7f540601d0c0 0 bucket checkpoint complete 2020-10-15T16:51:35.604 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-25 zones={test-zone2, test-zone1} 2020-10-15T16:51:35.615 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:35.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:35.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:35.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:35.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:35.732 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-25 zones={test-zone2, test-zone1} 2020-10-15T16:51:35.732 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-25 zones={test-zone3, test-zone1} 2020-10-15T16:51:35.741 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:35.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:35.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:35.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:35.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:35.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-25 zones={test-zone3, test-zone1} 2020-10-15T16:51:35.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-25 zones={test-zone3, test-zone2} 2020-10-15T16:51:35.868 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:35.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:35.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:35.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:35.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:35.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-25 zones={test-zone3, test-zone2} 2020-10-15T16:51:35.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:35.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:35.984 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:36.260 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:36.258+0000 7fd62ab410c0 1 bucket sync caught up with source: 2020-10-15T16:51:36.261 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.7.12, , ] 2020-10-15T16:51:36.261 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.7.12, , ] 2020-10-15T16:51:36.261 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:36.258+0000 7fd62ab410c0 0 bucket checkpoint complete 2020-10-15T16:51:36.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:36.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:36.289 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:36.337 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:36.337 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:36.572 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:36.564+0000 7f1771e900c0 1 bucket sync caught up with source: 2020-10-15T16:51:36.572 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.7.12, , ] 2020-10-15T16:51:36.573 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.7.12, , ] 2020-10-15T16:51:36.573 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:36.564+0000 7f1771e900c0 0 bucket checkpoint complete 2020-10-15T16:51:36.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:36.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:36.598 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:36.646 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:36.646 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:36.908 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:36.902+0000 7f4ee16cf0c0 1 bucket sync caught up with source: 2020-10-15T16:51:36.908 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000001.7.12, , ] 2020-10-15T16:51:36.909 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000001.7.12, , ] 2020-10-15T16:51:36.909 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:36.902+0000 7f4ee16cf0c0 0 bucket checkpoint complete 2020-10-15T16:51:37.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.020 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:37.069 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:37.069 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:37.298 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:37.292+0000 7fefcf9190c0 1 bucket sync caught up with source: 2020-10-15T16:51:37.299 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.12.12, , ] 2020-10-15T16:51:37.299 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.12.12, , ] 2020-10-15T16:51:37.299 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:37.292+0000 7fefcf9190c0 0 bucket checkpoint complete 2020-10-15T16:51:37.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.332 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:37.376 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:37.377 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:37.608 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:37.600+0000 7f5c849db0c0 1 bucket sync caught up with source: 2020-10-15T16:51:37.608 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000001.12.12, , ] 2020-10-15T16:51:37.609 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000001.12.12, , ] 2020-10-15T16:51:37.609 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:37.600+0000 7f5c849db0c0 0 bucket checkpoint complete 2020-10-15T16:51:37.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.640 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:37.732 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:37.732 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:37.964 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:37.956+0000 7f1a8a2d70c0 1 bucket sync caught up with source: 2020-10-15T16:51:37.965 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2020-10-15T16:51:37.965 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2020-10-15T16:51:37.965 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:51:37.956+0000 7f1a8a2d70c0 0 bucket checkpoint complete 2020-10-15T16:51:37.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T16:51:37.993 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T16:51:38.042 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:38.042 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:38.274 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:38.270+0000 7f3c46ec00c0 1 bucket sync caught up with source: 2020-10-15T16:51:38.275 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2020-10-15T16:51:38.275 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2020-10-15T16:51:38.275 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:38.270+0000 7f3c46ec00c0 0 bucket checkpoint complete 2020-10-15T16:51:38.308 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-26 zones={test-zone2, test-zone1} 2020-10-15T16:51:38.319 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:38.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:38.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:38.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:38.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:38.449 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-26 zones={test-zone2, test-zone1} 2020-10-15T16:51:38.450 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-26 zones={test-zone3, test-zone1} 2020-10-15T16:51:38.459 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:38.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:38.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:38.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:38.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:38.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-26 zones={test-zone3, test-zone1} 2020-10-15T16:51:38.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-26 zones={test-zone3, test-zone2} 2020-10-15T16:51:38.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:51:38.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:38.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:51:38.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-15T16:51:38.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-15T16:51:38.707 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-26 zones={test-zone3, test-zone2} 2020-10-15T16:51:38.708 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-10-15T16:51:38.708 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-27 2020-10-15T16:51:38.741 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-28 2020-10-15T16:51:38.799 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-29 2020-10-15T16:51:39.050 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-10-15T16:51:39.051 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-30 2020-10-15T16:51:39.094 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-31 2020-10-15T16:51:39.151 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-32 2020-10-15T16:51:39.426 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-10-15T16:51:39.427 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-33 2020-10-15T16:51:39.457 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-34 2020-10-15T16:51:39.529 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-35 2020-10-15T16:51:41.814 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-10-15T16:51:41.815 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-36 2020-10-15T16:51:41.850 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-37 2020-10-15T16:51:41.915 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-38 2020-10-15T16:51:41.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:41.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:41.971 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:42.006 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:42.006 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:42.214 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:51:42.253 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:51:42.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.253 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780701.944929_311.1", 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:41.944929Z" 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.255 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:42.255 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:42.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.255 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:42.255 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:42.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.256 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:51:42.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:51:42.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:42.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:42.259 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:42.301 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:42.301 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:42.578 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:42.579 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:42.579 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:42.579 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:42.579 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:42.579 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:42.580 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:42.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780701.944929_311.1", 2020-10-15T16:51:42.582 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.944929Z", 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:42.584 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:42.585 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:42.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:42.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:42.588 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:42.589 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:51:42.589 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:51:42.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:42.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:42.590 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:42.630 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:42.630 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:42.869 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:51:42.869 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:51:42.870 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:51:42.870 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:51:42.870 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:51:42.870 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:42.870 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:42.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:42.871 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.872 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780701.944929_311.1", 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:42.873 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.944929Z", 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:42.874 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:42.875 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:42.876 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:51:42.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:42.878 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:51:42.878 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:51:42.878 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:51:42.878 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:42.878 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:51:42.879 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:51:42.880 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:51:42.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:42.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:42.881 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:42.921 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:42.922 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:43.294 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:43.294 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:43.294 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:43.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:43.296 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:43.297 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780701.944929_311.1", 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.944929Z", 2020-10-15T16:51:43.298 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:43.299 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:43.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:43.301 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:43.302 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:43.303 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:43.303 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:43.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:43.303 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:43.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:51:43.559 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:51:43.552+0000 7fecc07a0700 -1 virtual int RGWDataChangesFIFO::push(int, ceph::real_time, const string&, ceph::buffer::v15_2_0::list&&): unable to push to FIFO: data_log.2: (22) Invalid argument 2020-10-15T16:51:43.559 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:51:43.552+0000 7fecc07a0700 -1 ERROR: failed writing data log 2020-10-15T16:51:43.653 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-10-15T16:51:43.654 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-39 2020-10-15T16:51:43.694 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-40 2020-10-15T16:51:43.776 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-41 2020-10-15T16:51:43.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:43.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:43.842 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:43.875 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:43.876 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:44.063 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=0c649eb8-3e99-4d22-800d-adf2e2823b92 2020-10-15T16:51:44.088 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:51:44.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.089 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:44.089 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:42:34.156484Z" 2020-10-15T16:51:44.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.089 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780703.807546_324.1", 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.807546Z" 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.091 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:44.091 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:51:44.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.091 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:51:44.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:51:44.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:44.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:44.094 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:44.418 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:44.418 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:44.419 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:44.420 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:44.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:44.422 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:44.422 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:44.422 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:44.422 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780703.807546_324.1", 2020-10-15T16:51:44.422 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:44.422 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:44.423 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:44.423 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.807546Z", 2020-10-15T16:51:44.423 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:44.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:44.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:44.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:44.424 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:44.425 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:44.426 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:44.426 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:44.426 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:44.426 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:44.426 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:44.426 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:44.427 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:44.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:44.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:44.427 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:44.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:44.428 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:44.428 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:44.428 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:44.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:44.428 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:44.429 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:51:44.430 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:51:44.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:44.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:44.431 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:44.472 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:44.473 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:44.727 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:51:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:51:44.727 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:51:44.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:51:44.729 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:51:44.730 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780703.807546_324.1", 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.807546Z", 2020-10-15T16:51:44.731 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:44.732 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:51:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:51:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:51:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:51:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:51:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:51:44.734 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:51:44.735 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:51:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:51:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:51:44.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:51:44.736 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:51:44.737 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:51:44.737 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:51:44.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:44.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:51:44.740 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:51:44.779 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:44.779 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:45.142 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:45.143 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:51:45.143 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:51:45.143 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:51:45.143 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:51:45.143 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:45.144 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:45.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780703.807546_324.1", 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:45.146 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.807546Z", 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:45.147 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:45.148 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:51:45.149 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:45.151 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:45.152 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:51:45.153 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-10-15T16:51:45.153 INFO:teuthology.orchestra.run:waiting for 300 2020-10-15T16:51:45.154 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:51:45.148+0000 7fee1bfff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18281) UID: 0 2020-10-15T16:51:45.154 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:51:45.148+0000 7fee37bfd100 -1 shutting down 2020-10-15T16:51:48.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-10-15T16:51:51.255 INFO:tasks.rgw.c1.client.1:Stopped 2020-10-15T16:51:51.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:51:51.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:51:51.256 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:51:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:51.327 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:51.321+0000 7f18528020c0 0 NOTICE: overriding master zone: 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:51:51.329 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:51.325+0000 7f18528020c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:51:51.329 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:51.325+0000 7f18528020c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:51:51.329 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:51.325+0000 7f18528020c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:51:51.330 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:51:51.325+0000 7f18528020c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:51:51.346 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:51.347 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:51:51.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:51:51.347 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:51:51.347 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:51:51.347 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:51:51.347 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:51:51.348 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:51:51.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:51:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:51:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:51:51.352 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:51.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:51:51.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:51:51.354 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:51:51.436 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:51:51.436 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:51:51.757 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:51:51.757 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:51:51.757 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 1, 2020-10-15T16:51:51.757 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "0c649eb8-3e99-4d22-800d-adf2e2823b92", 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [ 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780154.156484_109.1", 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780703.807546_324.1", 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: "" 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.758 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:51:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:51:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:51:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:51:51.764 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.765 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:51:51.766 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:51:51.767 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:51:51.768 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.769 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.770 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:51:51.771 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:51:51.771 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:51:51.771 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:51:51.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.771 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:51:51.783 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:51:51.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.783 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:51:51.783 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:51:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:51:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:51:51.787 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 3 2020-10-15T16:51:51.788 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:51:51.789 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-10-15T16:51:54.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-10-15T16:52:51.791 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-42 2020-10-15T16:52:51.870 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-43 2020-10-15T16:52:51.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:52:51.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:52:51.910 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:52:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:52:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:52:52.153 INFO:teuthology.orchestra.run.smithi038.stderr:No --period given, using current period=b84643c0-8302-4cfb-a70b-7af9afa8e893 2020-10-15T16:52:52.178 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:52:52.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:52:52.178 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:52:52.179 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "0.000000" 2020-10-15T16:52:52.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:52:52.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:52:52.179 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780771.905271_175.1", 2020-10-15T16:52:52.179 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:52:51.905271Z" 2020-10-15T16:52:52.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "0.000000" 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:52:52.180 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "0.000000" 2020-10-15T16:52:52.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:52:52.181 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:52:52.182 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-10-15T16:52:52.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:52:52.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:52:52.183 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:52:52.224 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:52:52.224 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:52:52.484 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:52:52.484 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 3 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.485 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:52:52.486 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 80, 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 3 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:52.487 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780771.905271_175.1", 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:52:52.488 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.905271Z", 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 3 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:52:52.489 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 3 2020-10-15T16:52:52.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:52:52.491 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 3 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.492 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:52:52.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.493 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:52:52.493 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 80, 2020-10-15T16:52:52.493 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 80 2020-10-15T16:52:52.493 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.493 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:52:52.494 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-10-15T16:52:52.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:52:52.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:52:52.496 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:52:52.574 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:52:52.575 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:52:52.600 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:52:52.594+0000 7f5779dd20c0 0 NOTICE: overriding master zone: 669d2e2a-b26c-4898-99f7-fa22d626fe93 2020-10-15T16:52:52.601 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:52:52.598+0000 7f5779dd20c0 0 failed reading obj info from .rgw.root:zone_info.669d2e2a-b26c-4898-99f7-fa22d626fe93: (2) No such file or directory 2020-10-15T16:52:52.601 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:52:52.598+0000 7f5779dd20c0 0 WARNING: could not read zone params for zone id=669d2e2a-b26c-4898-99f7-fa22d626fe93 name=test-zone2 2020-10-15T16:52:52.603 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:52:52.598+0000 7f5779dd20c0 0 failed reading obj info from .rgw.root:zone_info.e36fa986-91ea-474b-bd28-fd91c0368de2: (2) No such file or directory 2020-10-15T16:52:52.603 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:52:52.598+0000 7f5779dd20c0 0 WARNING: could not read zone params for zone id=e36fa986-91ea-474b-bd28-fd91c0368de2 name=test-zone4 2020-10-15T16:52:52.612 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:52:52.613 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:52:52.613 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:52:52.613 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-10-15T16:52:52.613 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "test-zone1.rgw.control", 2020-10-15T16:52:52.613 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-10-15T16:52:52.613 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "test-zone1.rgw.log", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-10-15T16:52:52.614 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:52:52.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:52:52.616 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: "compression_type": "random" 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:52.618 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:52.618 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:52:52.618 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-10-15T16:52:52.618 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:52:52.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:52:52.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:52:52.620 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:52:52.697 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:52:52.697 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:52:52.999 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:52:53.000 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:52:53.000 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 1, 2020-10-15T16:52:53.000 INFO:teuthology.orchestra.run.smithi183.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:52:53.000 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": [ 2020-10-15T16:52:53.000 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: "" 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: "period_map": { 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:52:53.001 INFO:teuthology.orchestra.run.smithi183.stdout: "zonegroups": [ 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:52:53.002 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.003 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone2", 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:52:53.004 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:52:53.006 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:52:53.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.008 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:52:53.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone4", 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.010 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "pubsub", 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "false", 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [ 2020-10-15T16:52:53.011 INFO:teuthology.orchestra.run.smithi183.stdout: "test-zone1" 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.012 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:52:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:52:53.014 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:52:53.014 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:52:53.014 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:53.014 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:53.014 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:52:53.034 INFO:teuthology.orchestra.run.smithi183.stdout: "short_zone_ids": [ 2020-10-15T16:52:53.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.034 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:52:53.034 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 757601199 2020-10-15T16:52:53.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.035 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:52:53.035 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 1274429389 2020-10-15T16:52:53.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 770630339 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 2077483514 2020-10-15T16:52:53.036 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: "period_config": { 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:52:53.037 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:52:53.038 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:52:53.039 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_name": "test-realm", 2020-10-15T16:52:53.040 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:52:53.040 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:52:53.041 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-10-15T16:53:53.042 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-44 2020-10-15T16:53:53.095 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-45 2020-10-15T16:53:53.171 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-10-15T16:53:53.172 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-10-15T16:53:53.174 INFO:tasks.rgw.c1.client.1:Started 2020-10-15T16:53:53.175 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8001 2020-10-15T16:53:53.192 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:53:53.192 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:53:53.196 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi183.front.sepia.ceph.com port 8001: Connection refused 2020-10-15T16:53:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-15T16:53:54.198 INFO:teuthology.orchestra.run.smithi183:> curl http://smithi183.front.sepia.ceph.com:8001 2020-10-15T16:53:54.213 INFO:teuthology.orchestra.run.smithi183.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-15T16:53:54.213 INFO:teuthology.orchestra.run.smithi183.stderr: Dload Upload Total Spent Left Speed 2020-10-15T16:53:54.220 INFO:teuthology.orchestra.run.smithi183.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-10-15T16:53:54.220 INFO:teuthology.orchestra.run.smithi183.stdout:anonymous 2020-10-15T16:53:54.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:54.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:54.222 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:53:54.308 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:53:54.309 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:53:54.514 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T16:53:54.548 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:53:54.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:54.548 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:53:54.548 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780833.143078_227.1", 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:53:53.143078Z" 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:54.549 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:53:54.550 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:53:54.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:54.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:54.550 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:53:54.550 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:53:54.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:54.551 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:53:54.551 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:53:54.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:54.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:54.553 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:53:54.595 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:53:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:53:54.862 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:53:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:53:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:53:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:53:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:53:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:53:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:53:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:53:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:53:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:53:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:53:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:53:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:53:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:53:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780833.143078_227.1", 2020-10-15T16:53:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:53:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.143078Z", 2020-10-15T16:53:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:53:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:53:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:53:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:53:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:53:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:53:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:53:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:53:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:53:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:53:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:53:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:53:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:53:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:53:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:53:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:54.880 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:53:54.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:53:54.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:53:54.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:54.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:54.885 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:53:54.925 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:53:54.925 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:53:55.171 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:53:55.171 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:53:55.171 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:53:55.171 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:53:55.172 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:53:55.172 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:53:55.172 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:53:55.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:55.172 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:53:55.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:53:55.173 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:53:55.174 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:53:55.175 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:53:55.175 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780833.143078_227.1", 2020-10-15T16:53:55.175 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:53:55.175 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:53:55.175 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.143078Z", 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:53:55.176 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:53:55.177 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:53:55.178 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:55.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:53:55.180 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:53:55.181 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:53:55.182 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:53:55.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:55.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:53:55.183 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:53:55.223 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:53:55.224 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:53:55.590 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:53:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:53:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:53:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:53:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:53:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:53:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:53:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:53:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:53:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:53:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780833.143078_227.1", 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.143078Z", 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:53:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:53:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:53:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:53:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:53:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:53:55.600 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:53:55.601 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:53:55.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-39 zones={test-zone2, test-zone1} 2020-10-15T16:53:55.625 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:55.675 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:55.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-39 zones={test-zone2, test-zone1} 2020-10-15T16:53:55.741 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-39 zones={test-zone3, test-zone1} 2020-10-15T16:53:55.761 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:55.810 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:55.872 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-39 zones={test-zone3, test-zone1} 2020-10-15T16:53:55.873 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-39 zones={test-zone1, test-zone2} 2020-10-15T16:53:55.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:55.934 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-39 zones={test-zone1, test-zone2} 2020-10-15T16:53:56.003 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-39 zones={test-zone3, test-zone2} 2020-10-15T16:53:56.013 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.062 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.130 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-39 zones={test-zone3, test-zone2} 2020-10-15T16:53:56.131 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-39 zones={test-zone1, test-zone3} 2020-10-15T16:53:56.141 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.256 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-39 zones={test-zone1, test-zone3} 2020-10-15T16:53:56.257 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-39 zones={test-zone2, test-zone3} 2020-10-15T16:53:56.267 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.314 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-39 zones={test-zone2, test-zone3} 2020-10-15T16:53:56.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-40 zones={test-zone2, test-zone1} 2020-10-15T16:53:56.393 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-40 zones={test-zone2, test-zone1} 2020-10-15T16:53:56.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-40 zones={test-zone3, test-zone1} 2020-10-15T16:53:56.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-40 zones={test-zone3, test-zone1} 2020-10-15T16:53:56.637 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-40 zones={test-zone1, test-zone2} 2020-10-15T16:53:56.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.759 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-40 zones={test-zone1, test-zone2} 2020-10-15T16:53:56.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-40 zones={test-zone3, test-zone2} 2020-10-15T16:53:56.769 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.818 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:56.881 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-40 zones={test-zone3, test-zone2} 2020-10-15T16:53:56.882 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-40 zones={test-zone1, test-zone3} 2020-10-15T16:53:56.894 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:56.946 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.012 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-40 zones={test-zone1, test-zone3} 2020-10-15T16:53:57.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-40 zones={test-zone2, test-zone3} 2020-10-15T16:53:57.026 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.074 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-40 zones={test-zone2, test-zone3} 2020-10-15T16:53:57.138 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-41 zones={test-zone2, test-zone1} 2020-10-15T16:53:57.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.268 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-41 zones={test-zone2, test-zone1} 2020-10-15T16:53:57.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-41 zones={test-zone3, test-zone1} 2020-10-15T16:53:57.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.389 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-41 zones={test-zone3, test-zone1} 2020-10-15T16:53:57.389 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-41 zones={test-zone1, test-zone2} 2020-10-15T16:53:57.400 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.446 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-41 zones={test-zone1, test-zone2} 2020-10-15T16:53:57.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-41 zones={test-zone3, test-zone2} 2020-10-15T16:53:57.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-41 zones={test-zone3, test-zone2} 2020-10-15T16:53:57.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-41 zones={test-zone1, test-zone3} 2020-10-15T16:53:57.646 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.761 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-41 zones={test-zone1, test-zone3} 2020-10-15T16:53:57.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-41 zones={test-zone2, test-zone3} 2020-10-15T16:53:57.773 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:57.885 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-41 zones={test-zone2, test-zone3} 2020-10-15T16:53:57.886 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-42 zones={test-zone2, test-zone1} 2020-10-15T16:53:57.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:57.946 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-42 zones={test-zone2, test-zone1} 2020-10-15T16:53:58.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-42 zones={test-zone3, test-zone1} 2020-10-15T16:53:58.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.066 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.128 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-42 zones={test-zone3, test-zone1} 2020-10-15T16:53:58.129 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-42 zones={test-zone1, test-zone2} 2020-10-15T16:53:58.143 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.254 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-42 zones={test-zone1, test-zone2} 2020-10-15T16:53:58.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-42 zones={test-zone3, test-zone2} 2020-10-15T16:53:58.265 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.372 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-42 zones={test-zone3, test-zone2} 2020-10-15T16:53:58.373 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-42 zones={test-zone1, test-zone3} 2020-10-15T16:53:58.384 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-42 zones={test-zone1, test-zone3} 2020-10-15T16:53:58.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-42 zones={test-zone2, test-zone3} 2020-10-15T16:53:58.500 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.617 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-42 zones={test-zone2, test-zone3} 2020-10-15T16:53:58.617 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-43 zones={test-zone2, test-zone1} 2020-10-15T16:53:58.627 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.674 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.737 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-43 zones={test-zone2, test-zone1} 2020-10-15T16:53:58.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-43 zones={test-zone3, test-zone1} 2020-10-15T16:53:58.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-43 zones={test-zone3, test-zone1} 2020-10-15T16:53:58.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-43 zones={test-zone1, test-zone2} 2020-10-15T16:53:58.868 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:58.914 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:58.978 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-43 zones={test-zone1, test-zone2} 2020-10-15T16:53:58.978 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-43 zones={test-zone3, test-zone2} 2020-10-15T16:53:58.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.038 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-43 zones={test-zone3, test-zone2} 2020-10-15T16:53:59.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-43 zones={test-zone1, test-zone3} 2020-10-15T16:53:59.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.154 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-43 zones={test-zone1, test-zone3} 2020-10-15T16:53:59.217 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-43 zones={test-zone2, test-zone3} 2020-10-15T16:53:59.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.274 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-43 zones={test-zone2, test-zone3} 2020-10-15T16:53:59.339 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-44 zones={test-zone2, test-zone1} 2020-10-15T16:53:59.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.461 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-44 zones={test-zone2, test-zone1} 2020-10-15T16:53:59.462 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-44 zones={test-zone3, test-zone1} 2020-10-15T16:53:59.472 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.581 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-44 zones={test-zone3, test-zone1} 2020-10-15T16:53:59.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-44 zones={test-zone1, test-zone2} 2020-10-15T16:53:59.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.710 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-44 zones={test-zone1, test-zone2} 2020-10-15T16:53:59.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-44 zones={test-zone3, test-zone2} 2020-10-15T16:53:59.722 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-44 zones={test-zone3, test-zone2} 2020-10-15T16:53:59.835 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-44 zones={test-zone1, test-zone3} 2020-10-15T16:53:59.847 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:53:59.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:53:59.957 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-44 zones={test-zone1, test-zone3} 2020-10-15T16:53:59.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-44 zones={test-zone2, test-zone3} 2020-10-15T16:53:59.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-44 zones={test-zone2, test-zone3} 2020-10-15T16:54:00.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-45 zones={test-zone2, test-zone1} 2020-10-15T16:54:00.092 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.138 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.200 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-45 zones={test-zone2, test-zone1} 2020-10-15T16:54:00.201 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-45 zones={test-zone3, test-zone1} 2020-10-15T16:54:00.211 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.258 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.321 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-45 zones={test-zone3, test-zone1} 2020-10-15T16:54:00.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-45 zones={test-zone1, test-zone2} 2020-10-15T16:54:00.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.378 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.441 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-45 zones={test-zone1, test-zone2} 2020-10-15T16:54:00.441 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-45 zones={test-zone3, test-zone2} 2020-10-15T16:54:00.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-45 zones={test-zone3, test-zone2} 2020-10-15T16:54:00.561 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-45 zones={test-zone1, test-zone3} 2020-10-15T16:54:00.573 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.622 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.684 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-45 zones={test-zone1, test-zone3} 2020-10-15T16:54:00.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-45 zones={test-zone2, test-zone3} 2020-10-15T16:54:00.695 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T16:54:00.742 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T16:54:00.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-45 zones={test-zone2, test-zone3} 2020-10-15T16:54:00.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:00.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:00.806 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:00.841 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:00.841 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:01.070 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:01.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.071 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780022.078737_52.1", 2020-10-15T16:54:01.071 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "user", 2020-10-15T16:54:01.071 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:01.072 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:40:22.078737453Z", 2020-10-15T16:54:01.072 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.072 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.072 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.072 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.074 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.074 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780022.087940_55.1", 2020-10-15T16:54:01.074 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "user", 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:40:22.087940982Z", 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.020200_56.1", 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:01.077 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.020200486Z", 2020-10-15T16:54:01.078 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.078 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.078 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.078 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.078 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.035923_61.1", 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.035923903Z", 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.080 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.082 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.037528_62.1", 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1", 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.037528840Z", 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.083 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.084 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.040794_63.1", 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.085 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1", 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.040794414Z", 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.086 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.063903_64.1", 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.063903600Z", 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.088 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.090 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.068188_65.1", 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.068188141Z", 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.091 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.092 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.092 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:01.092 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.092 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.094 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.094 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.070462_66.1", 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2", 2020-10-15T16:54:01.095 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.070462424Z", 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.096 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.074696_67.1", 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2", 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.074696732Z", 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.098 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.100 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.236399_68.1", 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.236399569Z", 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.246813_69.1", 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:01.103 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.246813559Z", 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.104 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.249178_70.1", 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3", 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.249178308Z", 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.106 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.107 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.269888_71.1", 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.108 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3", 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.269888245Z", 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.109 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.110 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.562533_72.1", 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.562533806Z", 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.111 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.113 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.569340_75.1", 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.569340542Z", 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.115 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.571388_76.1", 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4", 2020-10-15T16:54:01.116 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.571388447Z", 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.117 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.578288_77.1", 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4", 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.578288258Z", 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.119 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.120 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.602337_78.1", 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.121 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.602337755Z", 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.122 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.607439_79.1", 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.607439055Z", 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.124 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.125 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.613564_80.1", 2020-10-15T16:54:01.126 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5", 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.613564853Z", 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.127 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.619553_81.1", 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5", 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.619553277Z", 2020-10-15T16:54:01.129 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.131 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.677228_82.1", 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.677228758Z", 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.132 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.133 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.686417_83.1", 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.134 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.686417087Z", 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.135 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.691746_84.1", 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6", 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.691746539Z", 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.137 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.701238_85.1", 2020-10-15T16:54:01.139 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6", 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.701238667Z", 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.141 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.610666_86.1", 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.610666330Z", 2020-10-15T16:54:01.142 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:01.143 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.615079_91.1", 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.615079770Z", 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.145 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.146 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.617109_92.1", 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.147 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.617109940Z", 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.148 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.624177_93.1", 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.624177067Z", 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.150 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.151 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.646587_94.1", 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.152 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.646587696Z", 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.153 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.154 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.651089_95.1", 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.651089989Z", 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.155 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.653750_96.1", 2020-10-15T16:54:01.157 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.653750502Z", 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.659158_97.1", 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.659158286Z", 2020-10-15T16:54:01.160 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.162 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.705514_98.1", 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.705514298Z", 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.163 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.164 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.714153_99.1", 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:01.165 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.714153152Z", 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:01.166 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.716741_100.1", 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.716741822Z", 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.168 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.169 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.720290_101.1", 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:01.170 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.720290421Z", 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.171 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780153.973023_102.1", 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:33.973023863Z", 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.173 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.174 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "remove" 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780153.980103_105.1", 2020-10-15T16:54:01.175 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:33.980103075Z", 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.177 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.037669_106.1", 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.037669362Z", 2020-10-15T16:54:01.178 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "remove" 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.044552_107.1", 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.044552548Z", 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.151609_108.1", 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.151609693Z", 2020-10-15T16:54:01.183 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:01.184 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "remove" 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.185 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.156484_109.1", 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484486Z", 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:01.186 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.187 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.436554_75.1", 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:01.188 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.436554576Z", 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.189 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.441203_78.1", 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.441203519Z", 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.191 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.192 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.444010_79.1", 2020-10-15T16:54:01.193 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10", 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.444010714Z", 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.194 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.195 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.448982_80.1", 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10", 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.448982102Z", 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.196 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.198 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.471604_81.1", 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.471604415Z", 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.476178_82.1", 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.476178067Z", 2020-10-15T16:54:01.201 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:01.202 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.478035_83.1", 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11", 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.478035273Z", 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.204 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.205 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.483334_84.1", 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.206 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11", 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.483334680Z", 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.207 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.536455_85.1", 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.536455959Z", 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.209 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.210 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.547313_86.1", 2020-10-15T16:54:01.211 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.547313800Z", 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.212 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.549635_87.1", 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12", 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.549635460Z", 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.214 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.557492_88.1", 2020-10-15T16:54:01.216 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12", 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.557492671Z", 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.218 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.661175_92.1", 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.661175781Z", 2020-10-15T16:54:01.219 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:01.220 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.665404_96.1", 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.665404097Z", 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.222 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.223 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.667489_97.1", 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.224 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13", 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.667489966Z", 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.225 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.671455_98.1", 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13", 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.671455049Z", 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.227 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.228 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.693318_99.1", 2020-10-15T16:54:01.229 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.693318799Z", 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.230 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.231 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.697718_100.1", 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.697718647Z", 2020-10-15T16:54:01.232 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.234 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.700284_101.1", 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14", 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.700284160Z", 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.236 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.704804_102.1", 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14", 2020-10-15T16:54:01.237 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.704804291Z", 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:01.238 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.750755_103.1", 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.750755269Z", 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.240 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.241 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.759406_104.1", 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:01.242 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.759406919Z", 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.243 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.761217_105.1", 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15", 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.761217640Z", 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.246 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.765834_106.1", 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.247 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15", 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.765834376Z", 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.248 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.955052_107.1", 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.955052062Z", 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.250 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.959162_113.1", 2020-10-15T16:54:01.252 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.959162975Z", 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.254 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.961086_114.1", 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16", 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.961086719Z", 2020-10-15T16:54:01.255 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:01.256 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.257 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.965760_115.1", 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16", 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.965760332Z", 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.258 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.259 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.992057_116.1", 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.260 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.992057998Z", 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.261 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.998223_117.1", 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.998223727Z", 2020-10-15T16:54:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.265 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.000912_118.1", 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17", 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.000912498Z", 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.266 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.267 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.005187_119.1", 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17", 2020-10-15T16:54:01.268 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.005187083Z", 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:01.269 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.060953_120.1", 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.060953979Z", 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.271 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.272 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.067276_121.1", 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.273 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.067276061Z", 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.274 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.275 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.069062_122.1", 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18", 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.069062558Z", 2020-10-15T16:54:01.276 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:01.277 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.072845_123.1", 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18", 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.072845282Z", 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.279 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.280 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.093654_124.1", 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.281 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.093654456Z", 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.282 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.099372_125.1", 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.099372812Z", 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.284 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.285 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.104083_126.1", 2020-10-15T16:54:01.286 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.104083201Z", 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.287 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.288 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.110041_127.1", 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.110041428Z", 2020-10-15T16:54:01.289 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.291 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.123317_128.1", 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.123317028Z", 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.293 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.136755_129.1", 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:01.294 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.136755384Z", 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:01.295 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.296 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.296 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.296 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.296 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.237117_130.1", 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.237117211Z", 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.297 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.299 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.242376_146.1", 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.242376619Z", 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.300 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.301 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.244152_147.1", 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19", 2020-10-15T16:54:01.302 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.244152088Z", 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.303 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:01.304 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.304 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.304 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.304 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.304 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.305 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.249725_148.1", 2020-10-15T16:54:01.305 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.306 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19", 2020-10-15T16:54:01.306 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.249725487Z", 2020-10-15T16:54:01.306 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.306 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.306 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.307 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.307 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.307 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:01.307 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.308 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.308 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.308 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.308 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.309 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.258102_149.1", 2020-10-15T16:54:01.309 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.309 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:01.309 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.258102372Z", 2020-10-15T16:54:01.309 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.309 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.310 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:01.310 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.310 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.310 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:01.310 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.311 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.311 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.311 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.311 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.312 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.263968_150.1", 2020-10-15T16:54:01.312 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.312 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:01.312 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.263968538Z", 2020-10-15T16:54:01.313 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.313 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.313 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:01.313 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.314 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.314 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:01.314 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.314 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.315 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.315 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.315 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.316 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.774083_151.1", 2020-10-15T16:54:01.316 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.317 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:01.317 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.774083854Z", 2020-10-15T16:54:01.317 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.318 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.318 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.318 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.319 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.319 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.319 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.320 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.320 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.321 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.321 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.778075_162.1", 2020-10-15T16:54:01.322 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.322 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:01.322 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.778075584Z", 2020-10-15T16:54:01.323 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.323 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.323 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.324 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.324 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.324 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.325 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.325 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.325 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.325 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.326 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.326 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.779994_163.1", 2020-10-15T16:54:01.327 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.327 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20", 2020-10-15T16:54:01.327 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.779994982Z", 2020-10-15T16:54:01.327 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.327 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.328 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.328 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.328 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.328 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:01.329 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.329 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.329 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.329 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.329 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.330 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.783569_164.1", 2020-10-15T16:54:01.331 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.331 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20", 2020-10-15T16:54:01.331 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.783569181Z", 2020-10-15T16:54:01.331 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.332 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.332 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.332 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.332 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.333 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:01.333 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.333 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.333 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.333 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.333 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.334 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.334 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.334 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.334 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780573.133653_165.1", 2020-10-15T16:54:01.335 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.335 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:01.335 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:33.133653795Z", 2020-10-15T16:54:01.335 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.335 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.336 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.336 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.336 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.336 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.336 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.337 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.337 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.337 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.338 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.338 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.338 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780573.141127_169.1", 2020-10-15T16:54:01.339 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.339 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:01.339 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:33.141127542Z", 2020-10-15T16:54:01.339 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.340 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.340 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.340 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.341 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.341 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.341 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.342 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.342 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.342 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.343 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.343 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.344 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780638.077273_184.1", 2020-10-15T16:54:01.344 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.344 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:01.344 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:50:38.077273627Z", 2020-10-15T16:54:01.344 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.345 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.345 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.346 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.346 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.346 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.346 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.347 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.347 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.347 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780638.085516_186.1", 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:50:38.085516566Z", 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.349 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.349 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:01.349 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.350 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.350 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.350 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.350 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.351 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.950817_187.1", 2020-10-15T16:54:01.351 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.352 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:01.352 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.950817471Z", 2020-10-15T16:54:01.352 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.352 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.353 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.353 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.353 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.353 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:01.354 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.354 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.354 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.354 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.355 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.355 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.955895_191.1", 2020-10-15T16:54:01.356 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.356 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:01.356 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.955895323Z", 2020-10-15T16:54:01.356 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.356 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.357 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.357 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.357 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.357 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:01.358 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.358 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.358 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.358 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.359 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.359 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.957899_192.1", 2020-10-15T16:54:01.359 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.360 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21", 2020-10-15T16:54:01.360 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.957899082Z", 2020-10-15T16:54:01.360 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.360 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.361 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.361 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.361 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.361 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:01.362 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.362 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.362 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.364 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.963694_193.1", 2020-10-15T16:54:01.364 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.364 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21", 2020-10-15T16:54:01.365 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.963694041Z", 2020-10-15T16:54:01.365 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.365 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.365 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.366 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.366 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.366 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:01.367 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.367 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.367 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.369 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.989141_194.1", 2020-10-15T16:54:01.369 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.369 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:01.369 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.989141836Z", 2020-10-15T16:54:01.370 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.370 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.370 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.370 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.371 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.371 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:01.371 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.372 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.372 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.372 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.372 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.373 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.373 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.993921_195.1", 2020-10-15T16:54:01.373 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.373 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:01.374 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.993921692Z", 2020-10-15T16:54:01.374 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.374 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.374 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.374 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.375 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.375 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:01.375 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.376 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.376 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.376 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.377 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.377 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.996598_196.1", 2020-10-15T16:54:01.377 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.377 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22", 2020-10-15T16:54:01.377 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.996598844Z", 2020-10-15T16:54:01.378 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.378 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.378 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.378 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.379 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.379 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:01.379 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.380 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.380 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.380 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.381 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.381 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.001078_197.1", 2020-10-15T16:54:01.381 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.382 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22", 2020-10-15T16:54:01.382 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.001078013Z", 2020-10-15T16:54:01.382 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.382 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.382 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.382 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.383 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.383 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:01.383 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.384 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.384 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.046900_198.1", 2020-10-15T16:54:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.046900555Z", 2020-10-15T16:54:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.386 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.386 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.386 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.387 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.387 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.387 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:01.387 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.388 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.388 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.388 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.051294_199.1", 2020-10-15T16:54:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.051294115Z", 2020-10-15T16:54:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.392 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:01.392 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.392 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.392 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.393 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.393 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.393 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.393 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.393 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.394 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.053404_200.1", 2020-10-15T16:54:01.394 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.394 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23", 2020-10-15T16:54:01.394 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.053404083Z", 2020-10-15T16:54:01.395 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.395 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.395 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.395 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.396 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.396 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.396 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:01.396 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.398 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.398 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.057559_201.1", 2020-10-15T16:54:01.398 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.399 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23", 2020-10-15T16:54:01.399 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.057559943Z", 2020-10-15T16:54:01.399 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.399 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.399 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.400 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.400 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.400 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.400 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:01.400 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.401 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.401 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.401 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.401 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.402 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.402 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.078368_202.1", 2020-10-15T16:54:01.402 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.403 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:01.403 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.078368231Z", 2020-10-15T16:54:01.403 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.403 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.403 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:01.404 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.404 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.404 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.406 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.406 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.083939_203.1", 2020-10-15T16:54:01.406 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.406 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:01.407 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.083939714Z", 2020-10-15T16:54:01.407 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.407 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.407 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.409 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:01.409 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.409 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.410 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.410 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.410 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.410 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.411 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.088683_204.1", 2020-10-15T16:54:01.411 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.411 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:01.411 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.088683934Z", 2020-10-15T16:54:01.412 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.412 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.412 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:01.412 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.413 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.413 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:01.413 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.414 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.414 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.414 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.414 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.415 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.094969_205.1", 2020-10-15T16:54:01.415 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.415 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:01.415 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.094969549Z", 2020-10-15T16:54:01.415 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.416 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.416 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:01.416 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.416 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.417 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:01.417 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.417 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.418 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.418 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.418 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.419 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.419 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.419 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.106109_206.1", 2020-10-15T16:54:01.419 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.419 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:01.420 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.106109849Z", 2020-10-15T16:54:01.420 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.420 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.420 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:01.420 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.421 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.421 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.421 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:01.421 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.422 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.422 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.423 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.423 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.424 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.113514_207.1", 2020-10-15T16:54:01.424 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.424 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:01.425 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.113514114Z", 2020-10-15T16:54:01.425 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.425 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.425 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:01.426 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.426 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.426 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:01.427 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.427 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.427 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.428 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.428 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.428 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.428 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.429 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.522956_208.1", 2020-10-15T16:54:01.429 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.429 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:01.429 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.522956427Z", 2020-10-15T16:54:01.430 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.430 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.430 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.430 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.431 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.431 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.431 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.432 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.432 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.432 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.433 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.433 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.526974_216.1", 2020-10-15T16:54:01.434 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.434 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:01.434 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.526974067Z", 2020-10-15T16:54:01.434 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.435 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.435 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.435 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.436 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.436 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.436 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.437 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.437 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.437 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.438 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.438 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.529080_217.1", 2020-10-15T16:54:01.439 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.439 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24", 2020-10-15T16:54:01.439 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.529080378Z", 2020-10-15T16:54:01.440 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.440 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.440 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.440 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.441 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.441 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:01.441 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.442 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.442 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.443 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.532761_218.1", 2020-10-15T16:54:01.443 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.443 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24", 2020-10-15T16:54:01.444 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.532761600Z", 2020-10-15T16:54:01.444 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.444 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.444 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.444 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.445 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.445 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:01.445 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.446 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.446 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.446 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.446 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.447 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.558298_219.1", 2020-10-15T16:54:01.447 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.448 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:01.448 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.558298155Z", 2020-10-15T16:54:01.448 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.448 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.448 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.449 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.449 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.449 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.449 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.451 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.562423_220.1", 2020-10-15T16:54:01.451 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.451 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:01.452 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.562423620Z", 2020-10-15T16:54:01.452 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.452 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.452 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.452 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.453 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.453 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.453 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.454 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.454 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.456 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.564100_221.1", 2020-10-15T16:54:01.456 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.456 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25", 2020-10-15T16:54:01.456 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.564100877Z", 2020-10-15T16:54:01.457 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.457 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.457 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.457 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.458 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.458 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:01.458 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.459 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.459 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.459 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.459 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.460 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.567945_222.1", 2020-10-15T16:54:01.460 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.460 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25", 2020-10-15T16:54:01.461 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.567945059Z", 2020-10-15T16:54:01.461 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.461 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.461 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.461 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.462 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.462 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:01.462 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.463 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.463 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.463 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.463 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.464 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.612199_223.1", 2020-10-15T16:54:01.464 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.464 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:01.464 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.612199960Z", 2020-10-15T16:54:01.465 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.465 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.465 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.465 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.466 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.466 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.466 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.467 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.467 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.467 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.467 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.468 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.616295_224.1", 2020-10-15T16:54:01.468 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.468 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:01.468 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.616295813Z", 2020-10-15T16:54:01.468 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.469 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.469 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.469 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.470 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.470 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.470 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.471 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.471 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.472 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.473 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.618054_225.1", 2020-10-15T16:54:01.473 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.473 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26", 2020-10-15T16:54:01.474 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.618054466Z", 2020-10-15T16:54:01.474 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.474 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.474 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.475 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.475 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.475 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:01.476 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.476 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.476 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.477 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.477 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.478 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.622228_226.1", 2020-10-15T16:54:01.478 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.478 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26", 2020-10-15T16:54:01.478 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.622228203Z", 2020-10-15T16:54:01.479 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.479 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.479 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.479 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.480 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.480 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:01.480 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.481 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.481 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.482 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.482 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.640971_227.1", 2020-10-15T16:54:01.482 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.482 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:01.483 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.640971196Z", 2020-10-15T16:54:01.483 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.483 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.483 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.483 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.484 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.484 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.484 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.485 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.485 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.485 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.485 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.486 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.650996_228.1", 2020-10-15T16:54:01.486 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.486 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:01.487 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.650996655Z", 2020-10-15T16:54:01.487 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.487 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.487 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.487 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.488 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.488 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.488 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.489 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.489 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.491 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.654993_229.1", 2020-10-15T16:54:01.491 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.491 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:01.491 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.654993267Z", 2020-10-15T16:54:01.492 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.492 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.492 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.493 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.493 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.493 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.493 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.494 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.494 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.495 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.495 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.496 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.663850_230.1", 2020-10-15T16:54:01.496 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.496 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:01.496 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.663850522Z", 2020-10-15T16:54:01.496 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.497 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.497 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.497 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.498 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.498 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:01.498 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.499 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.499 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.499 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.499 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.500 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.670414_231.1", 2020-10-15T16:54:01.500 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.501 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:01.501 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.670414149Z", 2020-10-15T16:54:01.501 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.501 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.502 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.502 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.502 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.503 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.503 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.503 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.503 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.504 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.504 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.505 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.677960_232.1", 2020-10-15T16:54:01.505 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.505 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:01.505 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.677960563Z", 2020-10-15T16:54:01.506 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.506 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.506 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.506 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.507 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.507 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.507 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.508 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.508 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.509 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.509 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.689991_233.1", 2020-10-15T16:54:01.510 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.510 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:01.510 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.689991599Z", 2020-10-15T16:54:01.510 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.510 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.511 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.511 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.511 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.512 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.512 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.512 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.512 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.513 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.513 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.513 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.699262_234.1", 2020-10-15T16:54:01.514 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.514 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:01.514 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.699262282Z", 2020-10-15T16:54:01.515 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.515 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.515 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.515 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.516 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.516 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:01.516 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.518 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.717126_235.1", 2020-10-15T16:54:01.518 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.518 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:01.519 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.717126422Z", 2020-10-15T16:54:01.519 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.519 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.519 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.520 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.520 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.520 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.520 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.521 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.521 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.522 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.522 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.523 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.726779_236.1", 2020-10-15T16:54:01.523 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.523 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:01.524 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.726779903Z", 2020-10-15T16:54:01.524 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.524 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.524 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.525 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.525 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.525 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.526 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.526 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.526 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.527 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.739244_237.1", 2020-10-15T16:54:01.528 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.528 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:01.528 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.739244614Z", 2020-10-15T16:54:01.528 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.529 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.529 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.529 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.530 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.530 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.530 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.530 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.531 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.532 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.747532_238.1", 2020-10-15T16:54:01.532 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.533 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:01.533 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.747532325Z", 2020-10-15T16:54:01.533 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.533 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.534 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.534 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.534 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.535 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:01.535 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:01.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.535 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.536 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.725458_239.1", 2020-10-15T16:54:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.725458610Z", 2020-10-15T16:54:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.539 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.539 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.539 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:01.540 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.540 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.540 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.541 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.541 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.541 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.729952_247.1", 2020-10-15T16:54:01.542 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.542 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:01.542 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.729952098Z", 2020-10-15T16:54:01.543 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.543 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.543 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.543 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.544 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.544 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:01.544 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.545 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.545 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.545 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.545 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.546 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.732194_248.1", 2020-10-15T16:54:01.546 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.547 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27", 2020-10-15T16:54:01.547 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.732194555Z", 2020-10-15T16:54:01.547 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.547 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.548 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.548 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.548 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.548 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:01.549 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.549 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.549 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.550 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.550 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.551 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.736099_249.1", 2020-10-15T16:54:01.551 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.551 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27", 2020-10-15T16:54:01.551 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.736099973Z", 2020-10-15T16:54:01.551 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.552 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.552 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.552 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.553 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.553 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:01.553 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.554 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.554 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.555 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.763018_250.1", 2020-10-15T16:54:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.763018073Z", 2020-10-15T16:54:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.556 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.557 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.558 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.767193_251.1", 2020-10-15T16:54:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.767193540Z", 2020-10-15T16:54:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.560 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.561 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.562 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.769081_252.1", 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.563 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28", 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.769081040Z", 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.564 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.772622_253.1", 2020-10-15T16:54:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28", 2020-10-15T16:54:01.566 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.772622294Z", 2020-10-15T16:54:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.567 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.820914_254.1", 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:01.569 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.820914739Z", 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.570 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:01.571 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.571 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.571 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.571 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.571 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.828708_255.1", 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.828708805Z", 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.572 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.574 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.831143_256.1", 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29", 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.831143283Z", 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.575 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.836443_257.1", 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29", 2020-10-15T16:54:01.577 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.836443295Z", 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.578 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.861145_258.1", 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.861145621Z", 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.580 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.867657_259.1", 2020-10-15T16:54:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.867657187Z", 2020-10-15T16:54:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.916588_260.1", 2020-10-15T16:54:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.916588779Z", 2020-10-15T16:54:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.927095_261.1", 2020-10-15T16:54:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.927095157Z", 2020-10-15T16:54:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.987203_262.1", 2020-10-15T16:54:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.987203501Z", 2020-10-15T16:54:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.596 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.596 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.596 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.596 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.597 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.991971_263.1", 2020-10-15T16:54:01.597 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.597 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:01.597 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.991971647Z", 2020-10-15T16:54:01.598 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.598 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.598 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:01.598 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.599 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.599 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:01.599 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.600 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.600 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.601 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.072759_264.1", 2020-10-15T16:54:01.601 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.601 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:01.601 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.072759848Z", 2020-10-15T16:54:01.602 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.602 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.602 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.602 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.603 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.603 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:01.603 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.604 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.604 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.604 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.604 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.605 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.080572_265.1", 2020-10-15T16:54:01.605 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.605 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:01.606 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.080572799Z", 2020-10-15T16:54:01.606 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.606 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.606 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.607 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.607 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.607 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:01.607 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.608 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.608 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.609 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.083324_266.1", 2020-10-15T16:54:01.609 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.609 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30", 2020-10-15T16:54:01.609 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.083324055Z", 2020-10-15T16:54:01.610 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.610 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.611 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.611 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.611 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:01.611 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.612 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.612 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.612 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.612 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.613 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.087985_267.1", 2020-10-15T16:54:01.613 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.613 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30", 2020-10-15T16:54:01.614 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.087985505Z", 2020-10-15T16:54:01.614 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.614 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.614 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.615 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.615 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:01.616 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.617 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.115281_268.1", 2020-10-15T16:54:01.618 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.618 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:01.618 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.115281942Z", 2020-10-15T16:54:01.618 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.618 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.619 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.619 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.619 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.620 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:01.620 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.620 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.620 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.621 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.621 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.622 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.119612_269.1", 2020-10-15T16:54:01.622 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.622 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:01.622 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.119612435Z", 2020-10-15T16:54:01.623 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.623 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.623 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.623 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.624 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.624 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:01.624 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.625 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.625 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.626 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.122031_270.1", 2020-10-15T16:54:01.627 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.627 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31", 2020-10-15T16:54:01.627 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.122031946Z", 2020-10-15T16:54:01.627 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.627 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.628 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.628 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.628 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.628 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:01.629 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.629 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.629 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.629 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.630 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.126491_271.1", 2020-10-15T16:54:01.630 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.631 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31", 2020-10-15T16:54:01.631 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.126491865Z", 2020-10-15T16:54:01.631 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.631 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.631 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.632 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.632 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.632 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:01.632 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.633 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.633 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.633 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.633 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.634 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.170827_272.1", 2020-10-15T16:54:01.634 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.634 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:01.635 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.170827362Z", 2020-10-15T16:54:01.635 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.635 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.635 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.636 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.636 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.636 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:01.636 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.637 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.637 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.638 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.177992_273.1", 2020-10-15T16:54:01.638 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.639 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:01.639 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.177992630Z", 2020-10-15T16:54:01.639 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.639 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.639 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.640 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.640 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.640 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:01.641 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.641 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.641 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.642 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.188242_274.1", 2020-10-15T16:54:01.643 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.643 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32", 2020-10-15T16:54:01.643 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.188242974Z", 2020-10-15T16:54:01.643 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.644 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.644 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.644 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.645 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.645 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:01.645 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.646 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.646 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.646 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.646 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.647 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.193410_275.1", 2020-10-15T16:54:01.647 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.647 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32", 2020-10-15T16:54:01.647 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.193410108Z", 2020-10-15T16:54:01.648 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.648 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.648 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.648 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.649 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.649 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:01.649 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.650 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.650 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.651 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.228520_276.1", 2020-10-15T16:54:01.652 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.652 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:01.652 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.228520778Z", 2020-10-15T16:54:01.652 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.653 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.653 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:01.653 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.654 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.654 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:01.654 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.655 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.655 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.655 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.656 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.656 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.233934_277.1", 2020-10-15T16:54:01.656 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.657 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:01.657 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.233934127Z", 2020-10-15T16:54:01.657 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.657 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.658 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:01.658 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.658 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.659 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:01.659 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.660 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.660 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.660 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.660 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.661 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.285734_278.1", 2020-10-15T16:54:01.661 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.662 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:01.662 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.285734573Z", 2020-10-15T16:54:01.662 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.662 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.663 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:01.663 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.663 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.664 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:01.664 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.664 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.665 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.665 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.295813_279.1", 2020-10-15T16:54:01.666 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.666 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:01.666 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.295813271Z", 2020-10-15T16:54:01.666 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.667 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.667 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:01.667 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.668 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.668 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:01.668 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.668 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.669 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.670 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.358209_280.1", 2020-10-15T16:54:01.670 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.670 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:01.671 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.358209390Z", 2020-10-15T16:54:01.671 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.671 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.671 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:01.671 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.672 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.672 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:01.672 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.673 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.673 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.674 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.674 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.369099_281.1", 2020-10-15T16:54:01.674 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.675 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:01.675 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.369099338Z", 2020-10-15T16:54:01.675 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.676 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.676 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:01.676 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.676 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.677 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:01.677 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.678 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.678 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.679 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.440343_282.1", 2020-10-15T16:54:01.679 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.679 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:01.679 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.440343394Z", 2020-10-15T16:54:01.680 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.680 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.680 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.680 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.681 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.681 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:01.681 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.682 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.682 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.682 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.682 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.683 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.444859_283.1", 2020-10-15T16:54:01.683 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.683 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:01.684 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.444859665Z", 2020-10-15T16:54:01.684 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.684 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.684 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.684 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.685 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.685 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:01.685 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.686 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.686 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.687 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.446812_284.1", 2020-10-15T16:54:01.687 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.688 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33", 2020-10-15T16:54:01.688 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.446812694Z", 2020-10-15T16:54:01.688 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.688 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.689 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.689 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.689 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.689 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:01.690 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.690 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.691 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.691 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.451294_285.1", 2020-10-15T16:54:01.692 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.692 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33", 2020-10-15T16:54:01.692 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.451294634Z", 2020-10-15T16:54:01.692 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.693 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.693 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.693 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.694 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.694 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:01.694 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.694 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.695 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.696 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.478516_286.1", 2020-10-15T16:54:01.696 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.696 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:01.696 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.478516894Z", 2020-10-15T16:54:01.696 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.697 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.697 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.697 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.697 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.698 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:01.698 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.698 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.698 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.699 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.699 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.699 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.483226_287.1", 2020-10-15T16:54:01.700 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.700 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:01.700 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.483226132Z", 2020-10-15T16:54:01.700 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.701 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.701 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.701 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.702 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.702 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:01.702 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.703 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.703 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.704 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.485601_288.1", 2020-10-15T16:54:01.704 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.704 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34", 2020-10-15T16:54:01.705 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.485601115Z", 2020-10-15T16:54:01.705 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.705 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.705 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.706 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.706 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.706 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:01.706 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.707 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.707 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.709 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.492457_289.1", 2020-10-15T16:54:01.709 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.709 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34", 2020-10-15T16:54:01.709 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.492457380Z", 2020-10-15T16:54:01.710 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.710 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.710 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.710 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.711 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.711 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:01.711 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.712 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.712 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.713 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.713 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.553617_290.1", 2020-10-15T16:54:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.553617175Z", 2020-10-15T16:54:01.715 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.715 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.715 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.715 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.716 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.716 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:01.716 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.717 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.717 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.718 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.563843_291.1", 2020-10-15T16:54:01.719 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.719 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:01.719 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.563843665Z", 2020-10-15T16:54:01.719 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.719 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.720 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.720 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.720 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.721 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:01.721 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.722 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.722 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.722 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.722 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.723 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.568864_292.1", 2020-10-15T16:54:01.723 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.724 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35", 2020-10-15T16:54:01.724 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.568864246Z", 2020-10-15T16:54:01.724 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.724 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.725 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.725 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.725 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.726 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:01.726 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.726 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.727 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.727 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.727 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.728 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.584697_293.1", 2020-10-15T16:54:01.728 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.728 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35", 2020-10-15T16:54:01.729 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.584697168Z", 2020-10-15T16:54:01.729 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.729 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.729 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.729 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.730 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.730 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:01.730 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.731 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.731 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.733 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.607874_294.1", 2020-10-15T16:54:01.733 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.733 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:01.733 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.607874022Z", 2020-10-15T16:54:01.733 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.734 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.734 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:01.734 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.735 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.735 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:01.735 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.736 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.736 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.737 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.624681_295.1", 2020-10-15T16:54:01.737 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.738 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:01.738 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.624681194Z", 2020-10-15T16:54:01.738 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.738 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.739 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:01.739 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.739 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.740 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:01.740 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.740 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.741 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.742 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780700.672575_296.1", 2020-10-15T16:54:01.742 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.742 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:01.743 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:40.672575058Z", 2020-10-15T16:54:01.743 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.743 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.743 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:01.744 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.744 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.744 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:01.744 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780700.684269_297.1", 2020-10-15T16:54:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.747 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:01.747 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:40.684269312Z", 2020-10-15T16:54:01.747 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.747 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:01.749 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.749 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.749 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.751 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.744760_298.1", 2020-10-15T16:54:01.751 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.751 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:01.752 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.744760968Z", 2020-10-15T16:54:01.752 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.752 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.752 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:01.753 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.753 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.754 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:01.754 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.754 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.755 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.756 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.758273_299.1", 2020-10-15T16:54:01.756 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.756 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:01.756 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.758273297Z", 2020-10-15T16:54:01.757 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.757 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.757 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:01.757 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.758 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.758 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:01.758 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:01.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.759 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.759 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.759 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.760 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.834807_300.1", 2020-10-15T16:54:01.761 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.761 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:01.761 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.834807066Z", 2020-10-15T16:54:01.761 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.761 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.762 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.762 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.762 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.763 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:01.763 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.763 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.764 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.765 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.839075_301.1", 2020-10-15T16:54:01.765 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.765 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:01.766 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.839075159Z", 2020-10-15T16:54:01.766 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.766 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.766 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.767 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.767 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.767 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:01.768 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.768 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.768 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.769 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.769 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.770 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.841110_302.1", 2020-10-15T16:54:01.770 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.770 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36", 2020-10-15T16:54:01.771 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.841110279Z", 2020-10-15T16:54:01.771 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.771 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.771 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.772 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.772 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.772 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:01.773 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.773 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.773 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.773 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.774 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.774 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.845049_303.1", 2020-10-15T16:54:01.775 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.775 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36", 2020-10-15T16:54:01.775 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.845049758Z", 2020-10-15T16:54:01.776 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.776 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.776 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.776 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.777 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.777 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:01.777 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.778 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.778 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.778 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.778 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.779 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.877008_304.1", 2020-10-15T16:54:01.779 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.780 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:01.780 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.877008165Z", 2020-10-15T16:54:01.780 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.780 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.781 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.781 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.781 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.781 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:01.782 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.782 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.782 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.784 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.881213_305.1", 2020-10-15T16:54:01.784 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.784 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:01.784 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.881213458Z", 2020-10-15T16:54:01.785 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.785 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.785 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.785 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.786 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.786 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:01.786 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.787 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.787 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.787 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.788 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.788 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.883257_306.1", 2020-10-15T16:54:01.788 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.789 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37", 2020-10-15T16:54:01.789 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.883257547Z", 2020-10-15T16:54:01.789 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.789 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.790 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.790 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.790 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.790 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:01.791 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.791 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.791 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.792 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.792 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.793 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.887331_307.1", 2020-10-15T16:54:01.793 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.793 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37", 2020-10-15T16:54:01.793 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.887331707Z", 2020-10-15T16:54:01.793 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.794 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.794 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.794 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.794 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.795 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:01.795 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.795 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.796 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.796 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.796 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.797 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.934865_308.1", 2020-10-15T16:54:01.797 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.797 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:01.797 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.934865315Z", 2020-10-15T16:54:01.797 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.798 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.798 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.798 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.799 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.799 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:01.799 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.800 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.800 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.800 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.801 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.801 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.939408_309.1", 2020-10-15T16:54:01.802 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.802 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:01.802 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.939408181Z", 2020-10-15T16:54:01.802 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.802 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.803 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.803 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.803 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.804 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:01.804 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.804 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.805 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.805 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.805 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.806 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.941448_310.1", 2020-10-15T16:54:01.806 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.806 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38", 2020-10-15T16:54:01.806 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.941448054Z", 2020-10-15T16:54:01.807 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.807 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.807 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.807 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.808 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.808 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:01.808 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.809 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.809 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.809 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.810 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.810 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.944929_311.1", 2020-10-15T16:54:01.810 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.811 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38", 2020-10-15T16:54:01.811 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.944929850Z", 2020-10-15T16:54:01.811 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.811 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.811 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.812 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.812 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.813 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:01.813 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.813 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.814 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.814 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.814 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.815 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.668305_312.1", 2020-10-15T16:54:01.815 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.815 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:01.816 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.668305083Z", 2020-10-15T16:54:01.816 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.816 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.816 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.817 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.817 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.818 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:01.818 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.818 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.819 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.819 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.819 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.820 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.677868_314.1", 2020-10-15T16:54:01.820 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.821 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:01.821 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.677868663Z", 2020-10-15T16:54:01.821 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.821 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.821 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.822 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.822 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.823 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:01.823 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.823 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.823 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.824 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.824 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.824 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.682548_315.1", 2020-10-15T16:54:01.825 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.825 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39", 2020-10-15T16:54:01.825 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.682548949Z", 2020-10-15T16:54:01.825 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.826 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.826 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.826 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.827 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.827 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:01.827 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.827 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.828 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.828 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.828 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.829 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.688701_316.1", 2020-10-15T16:54:01.829 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.829 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39", 2020-10-15T16:54:01.830 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.688701616Z", 2020-10-15T16:54:01.830 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.830 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.830 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.831 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.831 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.831 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:01.832 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.832 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.832 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.834 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.718879_317.1", 2020-10-15T16:54:01.834 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.834 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:01.834 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.718879552Z", 2020-10-15T16:54:01.834 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.834 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.835 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.835 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.835 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.836 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:01.836 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.836 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.837 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.837 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.837 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.838 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.724158_318.1", 2020-10-15T16:54:01.838 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.838 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:01.838 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.724158775Z", 2020-10-15T16:54:01.839 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.839 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.839 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.839 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.840 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:01.840 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.841 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.841 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.841 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.841 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.842 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.726454_319.1", 2020-10-15T16:54:01.842 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.842 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40", 2020-10-15T16:54:01.843 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.726454595Z", 2020-10-15T16:54:01.843 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.843 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.843 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.843 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.844 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.844 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:01.844 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.845 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.845 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.845 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.845 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.846 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.731618_320.1", 2020-10-15T16:54:01.846 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.846 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40", 2020-10-15T16:54:01.847 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.731618705Z", 2020-10-15T16:54:01.847 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.847 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.847 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:01.849 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.849 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.850 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.851 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.794124_321.1", 2020-10-15T16:54:01.851 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.851 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:01.851 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.794124321Z", 2020-10-15T16:54:01.852 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.852 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.852 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.852 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.853 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.853 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:01.853 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.854 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.854 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.854 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.854 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.855 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.797914_322.1", 2020-10-15T16:54:01.855 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:01.855 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:01.856 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.797914867Z", 2020-10-15T16:54:01.856 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.856 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.857 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.858 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.858 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.858 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:01.859 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.859 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.859 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.859 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.860 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.860 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.800340_323.1", 2020-10-15T16:54:01.860 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.861 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41", 2020-10-15T16:54:01.861 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.800340181Z", 2020-10-15T16:54:01.861 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.861 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.862 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.862 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.862 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.863 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:01.863 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.863 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.864 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:01.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:01.865 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.807546_324.1", 2020-10-15T16:54:01.865 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:01.865 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41", 2020-10-15T16:54:01.866 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.807546577Z", 2020-10-15T16:54:01.866 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:01.866 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:01.866 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:01.867 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:01.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.867 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:01.867 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:01.868 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:01.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:01.868 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:01.868 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:01.869 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.869 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.869 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:01.869 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:01.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:01.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:01.873 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:01.919 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:01.919 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:02.161 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:02.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.162 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780022.078737_52.1", 2020-10-15T16:54:02.162 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "user", 2020-10-15T16:54:02.162 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:02.163 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.078737453Z", 2020-10-15T16:54:02.163 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.163 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.164 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.164 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.165 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:02.165 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.166 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.166 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.166 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.169 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780022.087940_55.1", 2020-10-15T16:54:02.169 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "user", 2020-10-15T16:54:02.169 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:02.170 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.087940982Z", 2020-10-15T16:54:02.170 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.170 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.170 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.171 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.171 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:02.172 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.172 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.172 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.172 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.173 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.173 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.020200_56.1", 2020-10-15T16:54:02.174 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:02.174 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.020200486Z", 2020-10-15T16:54:02.174 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.175 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.175 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.175 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.178 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.035923_61.1", 2020-10-15T16:54:02.178 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:02.178 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.035923903Z", 2020-10-15T16:54:02.178 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.179 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.179 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.179 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.180 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:02.180 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.180 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.181 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.182 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.037528_62.1", 2020-10-15T16:54:02.182 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1", 2020-10-15T16:54:02.182 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.037528840Z", 2020-10-15T16:54:02.183 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.183 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.183 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.184 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:02.184 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.184 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.185 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.185 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.185 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.186 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.040794_63.1", 2020-10-15T16:54:02.186 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1", 2020-10-15T16:54:02.186 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.040794414Z", 2020-10-15T16:54:02.187 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.187 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.187 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.188 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:02.188 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.189 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.189 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.190 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.063903_64.1", 2020-10-15T16:54:02.190 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.063903600Z", 2020-10-15T16:54:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.068188_65.1", 2020-10-15T16:54:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.068188141Z", 2020-10-15T16:54:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.201 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.202 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.070462_66.1", 2020-10-15T16:54:02.202 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.203 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2", 2020-10-15T16:54:02.203 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.070462424Z", 2020-10-15T16:54:02.203 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.204 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.204 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.204 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.205 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:02.205 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.206 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.206 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.207 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.074696_67.1", 2020-10-15T16:54:02.208 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.208 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2", 2020-10-15T16:54:02.208 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.074696732Z", 2020-10-15T16:54:02.209 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.209 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.209 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.211 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.211 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.236399_68.1", 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.236399569Z", 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.246813_69.1", 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.246813559Z", 2020-10-15T16:54:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.249178_70.1", 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3", 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.249178308Z", 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.269888_71.1", 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3", 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.269888245Z", 2020-10-15T16:54:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.562533_72.1", 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.562533806Z", 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.569340_75.1", 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:02.226 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.569340542Z", 2020-10-15T16:54:02.226 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.226 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.226 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.227 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.571388_76.1", 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4", 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.571388447Z", 2020-10-15T16:54:02.228 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.230 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.578288_77.1", 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4", 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.578288258Z", 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.231 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.232 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.602337_78.1", 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.602337755Z", 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:02.234 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.607439_79.1", 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.607439055Z", 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.236 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.237 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.613564_80.1", 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5", 2020-10-15T16:54:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.613564853Z", 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.619553_81.1", 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5", 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.619553277Z", 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.242 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.677228_82.1", 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.677228758Z", 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.244 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.686417_83.1", 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.686417087Z", 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.246 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.248 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.691746_84.1", 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6", 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.691746539Z", 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.249 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.701238_85.1", 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6", 2020-10-15T16:54:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.701238667Z", 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.610666_86.1", 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.610666330Z", 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.615079_91.1", 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.615079770Z", 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.617109_92.1", 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.617109940Z", 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.624177_93.1", 2020-10-15T16:54:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.624177067Z", 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.646587_94.1", 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.646587696Z", 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.651089_95.1", 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.651089989Z", 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.267 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.268 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.653750_96.1", 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.653750502Z", 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.659158_97.1", 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.659158286Z", 2020-10-15T16:54:02.272 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.274 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.705514_98.1", 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.705514298Z", 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.275 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.276 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.714153_99.1", 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:02.277 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.714153152Z", 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.716741_100.1", 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.716741822Z", 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.281 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.720290_101.1", 2020-10-15T16:54:02.282 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.720290421Z", 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.283 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780153.973023_102.1", 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:33.973023863Z", 2020-10-15T16:54:02.285 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "remove" 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780153.980103_105.1", 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:33.980103075Z", 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.288 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.289 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.037669_106.1", 2020-10-15T16:54:02.290 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.037669362Z", 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.291 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "remove" 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.044552_107.1", 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.044552548Z", 2020-10-15T16:54:02.293 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.295 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.151609_108.1", 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.151609693Z", 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "remove" 2020-10-15T16:54:02.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.156484_109.1", 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:02.298 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484486Z", 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.436554_75.1", 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.436554576Z", 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.441203_78.1", 2020-10-15T16:54:02.303 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.441203519Z", 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.304 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.305 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.444010_79.1", 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10", 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.444010714Z", 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.306 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.448982_80.1", 2020-10-15T16:54:02.308 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10", 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.448982102Z", 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.471604_81.1", 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.471604415Z", 2020-10-15T16:54:02.311 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:02.312 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.313 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.476178_82.1", 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.476178067Z", 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.314 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.315 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.478035_83.1", 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11", 2020-10-15T16:54:02.316 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.478035273Z", 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.483334_84.1", 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11", 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.483334680Z", 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.319 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.320 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.536455_85.1", 2020-10-15T16:54:02.321 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.536455959Z", 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.322 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.547313_86.1", 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.547313800Z", 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.549635_87.1", 2020-10-15T16:54:02.326 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12", 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.549635460Z", 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.557492_88.1", 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12", 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.557492671Z", 2020-10-15T16:54:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:02.330 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.661175_92.1", 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.661175781Z", 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.332 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.333 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.665404_96.1", 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:02.334 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.665404097Z", 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.335 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.667489_97.1", 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13", 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.667489966Z", 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.338 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.671455_98.1", 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.339 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13", 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.671455049Z", 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.340 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.693318_99.1", 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.693318799Z", 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.342 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.697718_100.1", 2020-10-15T16:54:02.344 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.697718647Z", 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.700284_101.1", 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14", 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.700284160Z", 2020-10-15T16:54:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.349 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.704804_102.1", 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14", 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.704804291Z", 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.350 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.750755_103.1", 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.750755269Z", 2020-10-15T16:54:02.352 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:02.353 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.759406_104.1", 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.759406919Z", 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.355 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.356 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.761217_105.1", 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15", 2020-10-15T16:54:02.357 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.761217640Z", 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.358 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.765834_106.1", 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15", 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.765834376Z", 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.361 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.955052_107.1", 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.362 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.955052062Z", 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.363 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.959162_113.1", 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.959162975Z", 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.365 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.366 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.367 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.372 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.961086_114.1", 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16", 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.961086719Z", 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.373 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.374 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.965760_115.1", 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16", 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.965760332Z", 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.376 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.992057_116.1", 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.992057998Z", 2020-10-15T16:54:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:02.379 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.998223_117.1", 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.998223727Z", 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.381 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.382 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.382 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:02.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.384 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.000912_118.1", 2020-10-15T16:54:02.384 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.384 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17", 2020-10-15T16:54:02.384 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.000912498Z", 2020-10-15T16:54:02.384 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.384 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.385 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.005187_119.1", 2020-10-15T16:54:02.386 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17", 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.005187083Z", 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.060953_120.1", 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.060953979Z", 2020-10-15T16:54:02.389 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.067276_121.1", 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.067276061Z", 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.393 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.069062_122.1", 2020-10-15T16:54:02.394 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18", 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.069062558Z", 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.395 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.072845_123.1", 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18", 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.072845282Z", 2020-10-15T16:54:02.397 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:02.398 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.093654_124.1", 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.093654456Z", 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.400 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.401 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.099372_125.1", 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.402 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.099372812Z", 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.104083_126.1", 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.104083201Z", 2020-10-15T16:54:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:02.406 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.407 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.110041_127.1", 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.110041428Z", 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:02.408 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.123317_128.1", 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.123317028Z", 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.136755_129.1", 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.136755384Z", 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.237117_130.1", 2020-10-15T16:54:02.415 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.237117211Z", 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.242376_146.1", 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:02.418 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.242376619Z", 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:02.419 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.244152_147.1", 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19", 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.244152088Z", 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.421 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.422 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.249725_148.1", 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.423 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19", 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.249725487Z", 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.424 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.258102_149.1", 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.258102372Z", 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.426 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.263968_150.1", 2020-10-15T16:54:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.263968538Z", 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.774083_151.1", 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.774083854Z", 2020-10-15T16:54:02.431 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.433 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.778075_162.1", 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.778075584Z", 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.434 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.435 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.779994_163.1", 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20", 2020-10-15T16:54:02.436 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.779994982Z", 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:02.437 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.783569_164.1", 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20", 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.783569181Z", 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.440 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780573.133653_165.1", 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.441 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:33.133653795Z", 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780573.141127_169.1", 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:33.141127542Z", 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.444 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780638.077273_184.1", 2020-10-15T16:54:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:50:38.077273627Z", 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780638.085516_186.1", 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:50:38.085516566Z", 2020-10-15T16:54:02.449 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:02.450 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.950817_187.1", 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.950817471Z", 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.955895_191.1", 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.955895323Z", 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.957899_192.1", 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21", 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.957899082Z", 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.458 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.963694_193.1", 2020-10-15T16:54:02.459 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21", 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.963694041Z", 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.460 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.989141_194.1", 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.989141836Z", 2020-10-15T16:54:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.993921_195.1", 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.993921692Z", 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.996598_196.1", 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22", 2020-10-15T16:54:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.996598844Z", 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.001078_197.1", 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22", 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.001078013Z", 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.046900_198.1", 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.046900555Z", 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.051294_199.1", 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.051294115Z", 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.053404_200.1", 2020-10-15T16:54:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23", 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.053404083Z", 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.057559_201.1", 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23", 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.057559943Z", 2020-10-15T16:54:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.078368_202.1", 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.078368231Z", 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.083939_203.1", 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.083939714Z", 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.088683_204.1", 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.088683934Z", 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.488 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.094969_205.1", 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.490 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.094969549Z", 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.491 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.106109_206.1", 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.106109849Z", 2020-10-15T16:54:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:02.494 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.495 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.113514_207.1", 2020-10-15T16:54:02.496 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.496 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:02.496 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.113514114Z", 2020-10-15T16:54:02.496 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.496 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.496 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.497 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.522956_208.1", 2020-10-15T16:54:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.522956427Z", 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.526974_216.1", 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.526974067Z", 2020-10-15T16:54:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.529080_217.1", 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24", 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.529080378Z", 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.504 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.505 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.532761_218.1", 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.506 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24", 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.532761600Z", 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.558298_219.1", 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.558298155Z", 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.509 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.510 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.562423_220.1", 2020-10-15T16:54:02.511 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.562423620Z", 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.564100_221.1", 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25", 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.564100877Z", 2020-10-15T16:54:02.514 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:02.515 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.567945_222.1", 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25", 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.567945059Z", 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.517 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.612199_223.1", 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.519 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.612199960Z", 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.616295_224.1", 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.616295813Z", 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.522 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.618054_225.1", 2020-10-15T16:54:02.524 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26", 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.618054466Z", 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.622228_226.1", 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26", 2020-10-15T16:54:02.527 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.622228203Z", 2020-10-15T16:54:02.528 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.528 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.528 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.528 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.528 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.640971_227.1", 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.640971196Z", 2020-10-15T16:54:02.530 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.531 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.532 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.650996_228.1", 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.650996655Z", 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.533 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.534 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.654993_229.1", 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:02.535 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.654993267Z", 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.536 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.663850_230.1", 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.663850522Z", 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.538 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.541 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.670414_231.1", 2020-10-15T16:54:02.541 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:02.541 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.670414149Z", 2020-10-15T16:54:02.541 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.542 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.542 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.542 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.542 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.543 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.543 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.543 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.543 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.677960_232.1", 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:02.544 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.677960563Z", 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.546 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.546 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.546 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.546 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.546 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.547 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.689991_233.1", 2020-10-15T16:54:02.547 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.547 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.689991599Z", 2020-10-15T16:54:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.550 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.550 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.699262_234.1", 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.699262282Z", 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.552 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.717126_235.1", 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.553 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.717126422Z", 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.554 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.726779_236.1", 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.726779903Z", 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.556 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.558 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.739244_237.1", 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.739244614Z", 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.559 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.747532_238.1", 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.561 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.747532325Z", 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.562 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.725458_239.1", 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.725458610Z", 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.564 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.566 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.729952_247.1", 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.729952098Z", 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.567 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.568 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.732194_248.1", 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27", 2020-10-15T16:54:02.569 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.732194555Z", 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.570 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.736099_249.1", 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27", 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.736099973Z", 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.572 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.763018_250.1", 2020-10-15T16:54:02.574 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.763018073Z", 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.767193_251.1", 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.767193540Z", 2020-10-15T16:54:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.578 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.578 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.578 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.769081_252.1", 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28", 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.769081040Z", 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.580 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.582 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.582 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.772622_253.1", 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28", 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.772622294Z", 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.583 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.584 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.820914_254.1", 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.585 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:02.586 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.820914739Z", 2020-10-15T16:54:02.586 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.586 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.586 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.586 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.828708_255.1", 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.828708805Z", 2020-10-15T16:54:02.588 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:02.589 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.590 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.590 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.831143_256.1", 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29", 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.831143283Z", 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.591 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.592 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.592 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.592 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.592 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:02.592 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.593 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.836443_257.1", 2020-10-15T16:54:02.594 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.594 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29", 2020-10-15T16:54:02.594 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.836443295Z", 2020-10-15T16:54:02.594 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.594 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.594 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.595 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.595 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.595 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:02.595 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.596 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.861145_258.1", 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.861145621Z", 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.867657_259.1", 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.867657187Z", 2020-10-15T16:54:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.602 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.916588_260.1", 2020-10-15T16:54:02.602 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.602 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.916588779Z", 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.927095_261.1", 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.927095157Z", 2020-10-15T16:54:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.987203_262.1", 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.987203501Z", 2020-10-15T16:54:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.991971_263.1", 2020-10-15T16:54:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.991971647Z", 2020-10-15T16:54:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.072759_264.1", 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.072759848Z", 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.080572_265.1", 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.080572799Z", 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.083324_266.1", 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30", 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.083324055Z", 2020-10-15T16:54:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.087985_267.1", 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30", 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.087985505Z", 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.623 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.624 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.624 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.624 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.115281_268.1", 2020-10-15T16:54:02.624 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.625 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:02.625 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.115281942Z", 2020-10-15T16:54:02.625 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.625 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.625 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.625 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.626 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.626 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:02.626 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.626 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.627 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.627 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.119612_269.1", 2020-10-15T16:54:02.628 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.628 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:02.628 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.119612435Z", 2020-10-15T16:54:02.628 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.628 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.628 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.629 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.630 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.630 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.122031_270.1", 2020-10-15T16:54:02.631 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.631 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31", 2020-10-15T16:54:02.631 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.122031946Z", 2020-10-15T16:54:02.631 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.631 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.631 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.632 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.126491_271.1", 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31", 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.126491865Z", 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.170827_272.1", 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.170827362Z", 2020-10-15T16:54:02.636 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.638 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.177992_273.1", 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.177992630Z", 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.639 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.640 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.188242_274.1", 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32", 2020-10-15T16:54:02.641 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.188242974Z", 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:02.643 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.643 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.643 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.193410_275.1", 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32", 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.193410108Z", 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.644 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.646 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.228520_276.1", 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.228520778Z", 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:02.647 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.648 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.233934_277.1", 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.649 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.233934127Z", 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.650 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.651 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.285734_278.1", 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.285734573Z", 2020-10-15T16:54:02.652 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.654 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.295813_279.1", 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.295813271Z", 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:02.655 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.656 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.358209_280.1", 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.657 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.358209390Z", 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.658 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.369099_281.1", 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.369099338Z", 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.660 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.662 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.440343_282.1", 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.440343394Z", 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.663 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.664 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.444859_283.1", 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:02.665 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.444859665Z", 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:02.666 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.446812_284.1", 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33", 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.446812694Z", 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.451294_285.1", 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33", 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.451294634Z", 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.671 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.478516_286.1", 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.478516894Z", 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.673 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.674 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.483226_287.1", 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.675 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.483226132Z", 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.676 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.677 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.485601_288.1", 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34", 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.485601115Z", 2020-10-15T16:54:02.678 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:02.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.680 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.492457_289.1", 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34", 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.492457380Z", 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.682 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.553617_290.1", 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.683 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.553617175Z", 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.684 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.686 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.563843_291.1", 2020-10-15T16:54:02.686 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.686 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:02.686 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.563843665Z", 2020-10-15T16:54:02.686 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:02.687 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.688 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.568864_292.1", 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35", 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.568864246Z", 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.689 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.690 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.691 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.584697_293.1", 2020-10-15T16:54:02.691 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35", 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.584697168Z", 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.694 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.607874_294.1", 2020-10-15T16:54:02.694 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.694 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.607874022Z", 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.695 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.624681_295.1", 2020-10-15T16:54:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.624681194Z", 2020-10-15T16:54:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:02.698 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780700.672575_296.1", 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:40.672575058Z", 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.700 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.701 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780700.684269_297.1", 2020-10-15T16:54:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:40.684269312Z", 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.704 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:02.704 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.704 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.704 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.744760_298.1", 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.705 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.744760968Z", 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.706 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.708 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.758273_299.1", 2020-10-15T16:54:02.708 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.708 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:02.708 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.758273297Z", 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.709 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.834807_300.1", 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.834807066Z", 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.711 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.712 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.712 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.712 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.712 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:02.712 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.839075_301.1", 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.839075159Z", 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.714 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.715 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.841110_302.1", 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36", 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.841110279Z", 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.845049_303.1", 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36", 2020-10-15T16:54:02.719 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.845049758Z", 2020-10-15T16:54:02.720 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.720 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.720 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.720 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.720 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.877008_304.1", 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.877008165Z", 2020-10-15T16:54:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:02.723 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.881213_305.1", 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.881213458Z", 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.726 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.727 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.727 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.883257_306.1", 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37", 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.883257547Z", 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.887331_307.1", 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37", 2020-10-15T16:54:02.730 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.887331707Z", 2020-10-15T16:54:02.731 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.731 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.731 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.731 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.731 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.934865_308.1", 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.934865315Z", 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.733 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.735 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.939408_309.1", 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.939408181Z", 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.736 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.738 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.941448_310.1", 2020-10-15T16:54:02.738 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.738 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38", 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.941448054Z", 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.739 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.944929_311.1", 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38", 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.944929850Z", 2020-10-15T16:54:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.668305_312.1", 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.668305083Z", 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.744 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.745 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.745 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:02.745 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.746 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.677868_314.1", 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.677868663Z", 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.747 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.748 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.749 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.682548_315.1", 2020-10-15T16:54:02.749 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.749 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39", 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.682548949Z", 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.750 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.688701_316.1", 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39", 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.688701616Z", 2020-10-15T16:54:02.752 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:02.753 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.718879_317.1", 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.718879552Z", 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.755 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.724158_318.1", 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:02.757 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.724158775Z", 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.758 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.726454_319.1", 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40", 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.726454595Z", 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.760 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.761 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.731618_320.1", 2020-10-15T16:54:02.762 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40", 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.731618705Z", 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.794124_321.1", 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.794124321Z", 2020-10-15T16:54:02.765 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:02.766 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.767 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.797914_322.1", 2020-10-15T16:54:02.768 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:02.768 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:02.768 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.797914867Z", 2020-10-15T16:54:02.768 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.768 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.769 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.769 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.770 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.770 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:02.770 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.770 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.800340_323.1", 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.771 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41", 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.800340181Z", 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.772 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:02.774 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.807546_324.1", 2020-10-15T16:54:02.774 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:02.774 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41", 2020-10-15T16:54:02.774 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.807546577Z", 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:02.775 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:02.776 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:02.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:02.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:02.779 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:02.816 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:02.817 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:03.044 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:03.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780022.078737_52.1", 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "user", 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:40:22.078737453Z", 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.045 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.046 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780022.087940_55.1", 2020-10-15T16:54:03.047 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "user", 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:40:22.087940982Z", 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.020200_56.1", 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:03.050 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.020200486Z", 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:03.051 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.035923_61.1", 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.035923903Z", 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.054 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.054 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.054 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:03.054 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.037528_62.1", 2020-10-15T16:54:03.055 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1", 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.037528840Z", 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.057 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.040794_63.1", 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-1", 2020-10-15T16:54:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.040794414Z", 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.059 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.063903_64.1", 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.063903600Z", 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.061 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.063 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.068188_65.1", 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.068188141Z", 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.064 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.065 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.065 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:03.065 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.065 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.067 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.068 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.070462_66.1", 2020-10-15T16:54:03.068 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.068 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2", 2020-10-15T16:54:03.068 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.070462424Z", 2020-10-15T16:54:03.068 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.070 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.070 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.070 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.070 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780145.074696_67.1", 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-2", 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:25.074696732Z", 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.071 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.072 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.072 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.072 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.072 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:03.072 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.073 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.236399_68.1", 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.236399569Z", 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.074 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.075 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.246813_69.1", 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.076 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.246813559Z", 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.077 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.078 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:03.078 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.078 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.078 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.079 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.249178_70.1", 2020-10-15T16:54:03.079 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.079 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3", 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.249178308Z", 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.080 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780148.269888_71.1", 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-3", 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:28.269888245Z", 2020-10-15T16:54:03.082 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.083 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.083 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.083 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.083 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.083 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.562533_72.1", 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.562533806Z", 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.085 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.569340_75.1", 2020-10-15T16:54:03.087 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.569340542Z", 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.571388_76.1", 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4", 2020-10-15T16:54:03.090 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.571388447Z", 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:03.091 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.578288_77.1", 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-4", 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.578288258Z", 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.093 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.094 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.602337_78.1", 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.095 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.602337755Z", 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.096 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.097 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.607439_79.1", 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.607439055Z", 2020-10-15T16:54:03.098 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:03.099 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.100 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.100 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.100 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.100 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.613564_80.1", 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5", 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.613564853Z", 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.101 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.103 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.103 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.619553_81.1", 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-5", 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.619553277Z", 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.104 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.105 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.106 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.106 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.106 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.677228_82.1", 2020-10-15T16:54:03.106 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.677228758Z", 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.108 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.109 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.686417_83.1", 2020-10-15T16:54:03.109 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.109 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:03.109 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.686417087Z", 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:03.110 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.112 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.691746_84.1", 2020-10-15T16:54:03.112 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.112 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6", 2020-10-15T16:54:03.112 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.691746539Z", 2020-10-15T16:54:03.112 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.112 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.114 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780149.701238_85.1", 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-6", 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:29.701238667Z", 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.115 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.116 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.117 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.117 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.117 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.610666_86.1", 2020-10-15T16:54:03.117 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.118 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:03.118 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.610666330Z", 2020-10-15T16:54:03.118 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.118 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.118 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.118 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.119 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.119 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:03.119 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.119 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.615079_91.1", 2020-10-15T16:54:03.120 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.121 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:03.121 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.615079770Z", 2020-10-15T16:54:03.121 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.121 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.121 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.121 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.122 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.122 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:03.122 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.122 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.617109_92.1", 2020-10-15T16:54:03.123 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.124 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:03.124 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.617109940Z", 2020-10-15T16:54:03.124 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.124 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.124 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.124 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.125 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.624177_93.1", 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:03.126 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.624177067Z", 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.127 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.128 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.128 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.128 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.646587_94.1", 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.646587696Z", 2020-10-15T16:54:03.129 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:03.130 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.651089_95.1", 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.651089989Z", 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.132 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.133 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.133 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:03.133 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.653750_96.1", 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.653750502Z", 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.135 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.136 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.659158_97.1", 2020-10-15T16:54:03.137 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.659158286Z", 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.705514_98.1", 2020-10-15T16:54:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.705514298Z", 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.141 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.142 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:03.142 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.142 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.142 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.714153_99.1", 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.143 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.714153152Z", 2020-10-15T16:54:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.145 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.716741_100.1", 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.146 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.716741822Z", 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.147 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.149 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780152.720290_101.1", 2020-10-15T16:54:03.149 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.149 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:03.149 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:32.720290421Z", 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.150 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780153.973023_102.1", 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:33.973023863Z", 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.152 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.153 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "remove" 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780153.980103_105.1", 2020-10-15T16:54:03.154 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-7", 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:33.980103075Z", 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.037669_106.1", 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:03.157 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.037669362Z", 2020-10-15T16:54:03.158 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.158 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.158 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:03.158 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.158 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "remove" 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.044552_107.1", 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-8", 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.044552548Z", 2020-10-15T16:54:03.160 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.161 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.161 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:03.161 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.161 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.161 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.151609_108.1", 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.151609693Z", 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "remove" 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.165 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780154.156484_109.1", 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-9", 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484486Z", 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:03.166 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.167 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.167 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:03.167 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.167 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.168 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.168 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.436554_75.1", 2020-10-15T16:54:03.169 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.169 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:03.169 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.436554576Z", 2020-10-15T16:54:03.169 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.169 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.169 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.170 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.170 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.170 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:03.170 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.171 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.171 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.171 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.171 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.441203_78.1", 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.441203519Z", 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.172 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.173 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.444010_79.1", 2020-10-15T16:54:03.174 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10", 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.444010714Z", 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.175 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.448982_80.1", 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-10", 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.448982102Z", 2020-10-15T16:54:03.177 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.178 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.178 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.178 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.178 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.471604_81.1", 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.471604415Z", 2020-10-15T16:54:03.180 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:03.181 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.182 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.182 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.183 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.476178_82.1", 2020-10-15T16:54:03.183 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.183 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:03.183 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.476178067Z", 2020-10-15T16:54:03.183 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.184 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.184 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.184 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.184 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.184 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:03.185 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.185 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.185 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.186 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.478035_83.1", 2020-10-15T16:54:03.186 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.186 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11", 2020-10-15T16:54:03.186 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.478035273Z", 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.187 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.483334_84.1", 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-11", 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.483334680Z", 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.189 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.191 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.536455_85.1", 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.536455959Z", 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.192 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.193 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.547313_86.1", 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:03.194 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.547313800Z", 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.195 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.549635_87.1", 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12", 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.549635460Z", 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.197 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.198 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.198 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.198 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.198 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:03.198 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.199 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.199 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.199 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.199 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.200 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780155.557492_88.1", 2020-10-15T16:54:03.200 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.200 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-12", 2020-10-15T16:54:03.200 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:35.557492671Z", 2020-10-15T16:54:03.200 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:03.201 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.202 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.202 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.202 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.202 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.661175_92.1", 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.661175781Z", 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.203 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.205 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.665404_96.1", 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.665404097Z", 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.206 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.207 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.667489_97.1", 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13", 2020-10-15T16:54:03.208 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.667489966Z", 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.209 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.671455_98.1", 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-13", 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.671455049Z", 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.211 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.212 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.212 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.212 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.212 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:03.212 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.214 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.693318_99.1", 2020-10-15T16:54:03.214 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.214 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:03.214 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.693318799Z", 2020-10-15T16:54:03.214 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:03.215 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.216 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.216 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.216 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.216 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.217 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.697718_100.1", 2020-10-15T16:54:03.217 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.217 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:03.217 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.697718647Z", 2020-10-15T16:54:03.217 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:03.218 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.700284_101.1", 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14", 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.700284160Z", 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.220 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.221 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.704804_102.1", 2020-10-15T16:54:03.222 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-14", 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.704804291Z", 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.223 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.224 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.750755_103.1", 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.750755269Z", 2020-10-15T16:54:03.225 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.226 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.226 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.226 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.226 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.226 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:03.227 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.227 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.227 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.228 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.759406_104.1", 2020-10-15T16:54:03.228 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.228 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:03.228 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.759406919Z", 2020-10-15T16:54:03.229 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.229 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.229 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.229 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.761217_105.1", 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.231 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15", 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.761217640Z", 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.232 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.233 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780166.765834_106.1", 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-15", 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:46.765834376Z", 2020-10-15T16:54:03.234 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:03.235 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.236 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.955052_107.1", 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.955052062Z", 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.237 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.238 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.239 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.239 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.239 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.959162_113.1", 2020-10-15T16:54:03.239 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.959162975Z", 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.241 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.241 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:03.241 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.241 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.241 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.242 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.242 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.242 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.961086_114.1", 2020-10-15T16:54:03.242 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16", 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.961086719Z", 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.965760_115.1", 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.245 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16", 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.965760332Z", 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.246 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.992057_116.1", 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.992057998Z", 2020-10-15T16:54:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:03.249 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780215.998223_117.1", 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:35.998223727Z", 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.251 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.252 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.252 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.252 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:03.252 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.253 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.253 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.253 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.253 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.254 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.000912_118.1", 2020-10-15T16:54:03.254 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.254 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17", 2020-10-15T16:54:03.254 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.000912498Z", 2020-10-15T16:54:03.254 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.254 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.256 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.005187_119.1", 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17", 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.005187083Z", 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.257 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.258 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.060953_120.1", 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:03.259 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.060953979Z", 2020-10-15T16:54:03.260 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.260 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.260 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.260 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.260 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.261 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.262 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.067276_121.1", 2020-10-15T16:54:03.262 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.262 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:03.262 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.067276061Z", 2020-10-15T16:54:03.263 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.263 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.263 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.263 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.263 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.069062_122.1", 2020-10-15T16:54:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18", 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.069062558Z", 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.266 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.267 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:03.267 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.267 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.267 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.267 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.268 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.268 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.072845_123.1", 2020-10-15T16:54:03.268 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.268 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18", 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.072845282Z", 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.269 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.093654_124.1", 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.093654456Z", 2020-10-15T16:54:03.271 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.273 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.099372_125.1", 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.099372812Z", 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:03.274 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.275 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.104083_126.1", 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.276 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.104083201Z", 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.277 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.110041_127.1", 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.110041428Z", 2020-10-15T16:54:03.279 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:03.280 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.282 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.123317_128.1", 2020-10-15T16:54:03.282 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.282 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:03.282 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.123317028Z", 2020-10-15T16:54:03.282 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.282 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.283 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:03.283 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.283 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.283 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:03.283 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780216.136755_129.1", 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:43:36.136755384Z", 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:03.285 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.286 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.286 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:03.286 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.286 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.287 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.287 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.287 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.287 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.237117_130.1", 2020-10-15T16:54:03.288 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.288 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:03.288 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.237117211Z", 2020-10-15T16:54:03.288 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.288 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.288 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.289 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.290 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.290 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.290 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.290 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.242376_146.1", 2020-10-15T16:54:03.291 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.291 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:03.291 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.242376619Z", 2020-10-15T16:54:03.291 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.291 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.291 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.292 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.292 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.292 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:03.292 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.244152_147.1", 2020-10-15T16:54:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.294 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19", 2020-10-15T16:54:03.294 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.244152088Z", 2020-10-15T16:54:03.294 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.294 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.294 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.295 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.295 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.295 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:03.295 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.249725_148.1", 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19", 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.249725487Z", 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.297 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.298 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.258102_149.1", 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.299 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.258102372Z", 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.300 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780409.263968_150.1", 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:49.263968538Z", 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.302 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.303 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.774083_151.1", 2020-10-15T16:54:03.304 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.774083854Z", 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.305 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.306 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.778075_162.1", 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.778075584Z", 2020-10-15T16:54:03.307 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.308 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.309 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.779994_163.1", 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20", 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.779994982Z", 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.310 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.311 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780568.783569_164.1", 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20", 2020-10-15T16:54:03.312 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:28.783569181Z", 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.313 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780573.133653_165.1", 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:33.133653795Z", 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.315 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.316 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780573.141127_169.1", 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.317 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:49:33.141127542Z", 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.318 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.319 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780638.077273_184.1", 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:50:38.077273627Z", 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.320 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.322 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780638.085516_186.1", 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:50:38.085516566Z", 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.324 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.950817_187.1", 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.950817471Z", 2020-10-15T16:54:03.325 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:03.326 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.327 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.955895_191.1", 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.955895323Z", 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.328 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.329 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.957899_192.1", 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.330 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21", 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.957899082Z", 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.331 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.963694_193.1", 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21", 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.963694041Z", 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.334 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.989141_194.1", 2020-10-15T16:54:03.335 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.989141836Z", 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.336 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.337 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.993921_195.1", 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.993921692Z", 2020-10-15T16:54:03.338 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:03.339 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.340 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780677.996598_196.1", 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22", 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:17.996598844Z", 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.341 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.342 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.001078_197.1", 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22", 2020-10-15T16:54:03.343 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.001078013Z", 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.344 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.046900_198.1", 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.046900555Z", 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.346 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.347 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.051294_199.1", 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.051294115Z", 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.053404_200.1", 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23", 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.053404083Z", 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.351 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.352 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.057559_201.1", 2020-10-15T16:54:03.353 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23", 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.057559943Z", 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.354 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.355 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.078368_202.1", 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.078368231Z", 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.356 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.083939_203.1", 2020-10-15T16:54:03.358 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.083939714Z", 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.360 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.088683_204.1", 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.088683934Z", 2020-10-15T16:54:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.363 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.094969_205.1", 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.094969549Z", 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:03.364 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.365 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.106109_206.1", 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:03.366 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.106109849Z", 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:03.367 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780678.113514_207.1", 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:18.113514114Z", 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:03.369 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.370 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.522956_208.1", 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.371 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.522956427Z", 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.372 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.526974_216.1", 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.526974067Z", 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.374 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.375 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.529080_217.1", 2020-10-15T16:54:03.376 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24", 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.529080378Z", 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.377 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.378 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.532761_218.1", 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24", 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.532761600Z", 2020-10-15T16:54:03.379 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:03.380 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.381 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.558298_219.1", 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.558298155Z", 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.382 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.383 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.562423_220.1", 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:03.384 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.562423620Z", 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.385 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.386 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.564100_221.1", 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25", 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.564100877Z", 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.387 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.388 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.567945_222.1", 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.389 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25", 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.567945059Z", 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.390 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.391 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.612199_223.1", 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.612199960Z", 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.393 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.616295_224.1", 2020-10-15T16:54:03.394 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.616295813Z", 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.395 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.396 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.618054_225.1", 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26", 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.618054466Z", 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.397 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.622228_226.1", 2020-10-15T16:54:03.399 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26", 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.622228203Z", 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.400 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.401 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.640971_227.1", 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.640971196Z", 2020-10-15T16:54:03.402 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.404 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.650996_228.1", 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.650996655Z", 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.405 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.406 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.654993_229.1", 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:03.407 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.654993267Z", 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.408 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.663850_230.1", 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.663850522Z", 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.410 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.411 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.670414_231.1", 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.412 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.670414149Z", 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.413 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.414 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.677960_232.1", 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.677960563Z", 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.415 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.416 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.689991_233.1", 2020-10-15T16:54:03.417 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.689991599Z", 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.418 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.419 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.699262_234.1", 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.699262282Z", 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.420 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.421 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.717126_235.1", 2020-10-15T16:54:03.422 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.717126422Z", 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.424 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.726779_236.1", 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.726779903Z", 2020-10-15T16:54:03.425 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.426 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.427 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.739244_237.1", 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.739244614Z", 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.428 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.429 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780688.747532_238.1", 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:03.430 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:28.747532325Z", 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:03.431 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 3 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.725458_239.1", 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.725458610Z", 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.433 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.434 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.729952_247.1", 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:03.435 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.729952098Z", 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.436 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.732194_248.1", 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27", 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.732194555Z", 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.439 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.736099_249.1", 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.440 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27", 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.736099973Z", 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.441 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.763018_250.1", 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.763018073Z", 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.443 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.767193_251.1", 2020-10-15T16:54:03.445 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.767193540Z", 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.447 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.769081_252.1", 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28", 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.769081040Z", 2020-10-15T16:54:03.448 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:03.449 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.450 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.772622_253.1", 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28", 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.772622294Z", 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.451 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.452 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.820914_254.1", 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:03.453 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.820914739Z", 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.454 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.828708_255.1", 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.828708805Z", 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.456 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.457 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.831143_256.1", 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.458 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29", 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.831143283Z", 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.836443_257.1", 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29", 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.836443295Z", 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.462 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.861145_258.1", 2020-10-15T16:54:03.463 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.861145621Z", 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.464 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.465 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.867657_259.1", 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.867657187Z", 2020-10-15T16:54:03.466 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.468 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.916588_260.1", 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.916588779Z", 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:03.469 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.470 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.927095_261.1", 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.927095157Z", 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.472 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.987203_262.1", 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.987203501Z", 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.474 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.475 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780698.991971_263.1", 2020-10-15T16:54:03.476 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:38.991971647Z", 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.072759_264.1", 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.072759848Z", 2020-10-15T16:54:03.479 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.481 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.080572_265.1", 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.080572799Z", 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.483 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.083324_266.1", 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30", 2020-10-15T16:54:03.484 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.083324055Z", 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.485 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.487 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.087985_267.1", 2020-10-15T16:54:03.487 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.487 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30", 2020-10-15T16:54:03.487 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.087985505Z", 2020-10-15T16:54:03.487 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:03.488 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.489 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.489 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.490 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.115281_268.1", 2020-10-15T16:54:03.490 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.490 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:03.490 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.115281942Z", 2020-10-15T16:54:03.491 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.491 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.491 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.491 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.492 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.492 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:03.492 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.492 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.493 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.493 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.493 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.494 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.119612_269.1", 2020-10-15T16:54:03.494 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.494 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:03.494 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.119612435Z", 2020-10-15T16:54:03.494 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.494 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.495 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.122031_270.1", 2020-10-15T16:54:03.496 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31", 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.122031946Z", 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.498 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.126491_271.1", 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31", 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.126491865Z", 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.499 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.170827_272.1", 2020-10-15T16:54:03.501 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.170827362Z", 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.177992_273.1", 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.177992630Z", 2020-10-15T16:54:03.504 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.506 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.188242_274.1", 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32", 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.188242974Z", 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.193410_275.1", 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32", 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.193410108Z", 2020-10-15T16:54:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:03.510 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.511 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.228520_276.1", 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.228520778Z", 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:03.512 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.513 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.233934_277.1", 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:03.514 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.233934127Z", 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.515 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.285734_278.1", 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.285734573Z", 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.517 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.518 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.295813_279.1", 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.519 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.295813271Z", 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.520 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.358209_280.1", 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.358209390Z", 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.522 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.523 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.369099_281.1", 2020-10-15T16:54:03.524 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.369099338Z", 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.525 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.526 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.440343_282.1", 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.440343394Z", 2020-10-15T16:54:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.529 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.444859_283.1", 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.444859665Z", 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.530 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.446812_284.1", 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33", 2020-10-15T16:54:03.532 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.446812694Z", 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:03.533 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.451294_285.1", 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33", 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.451294634Z", 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.535 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.478516_286.1", 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:03.537 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.478516894Z", 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.538 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.483226_287.1", 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.483226132Z", 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.540 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.541 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.485601_288.1", 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.542 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34", 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.485601115Z", 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.543 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.492457_289.1", 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34", 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.492457380Z", 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.545 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.546 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.553617_290.1", 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.547 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.553617175Z", 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.548 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.563843_291.1", 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.563843665Z", 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.550 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.568864_292.1", 2020-10-15T16:54:03.552 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35", 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.568864246Z", 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.584697_293.1", 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35", 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.584697168Z", 2020-10-15T16:54:03.555 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.557 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.607874_294.1", 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.607874022Z", 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:03.558 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780699.624681_295.1", 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:03.560 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:39.624681194Z", 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:03.561 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780700.672575_296.1", 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:40.672575058Z", 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.563 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.564 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780700.684269_297.1", 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.565 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:40.684269312Z", 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.566 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.744760_298.1", 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.744760968Z", 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.568 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.569 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.758273_299.1", 2020-10-15T16:54:03.570 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.758273297Z", 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 2 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.834807_300.1", 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.834807066Z", 2020-10-15T16:54:03.573 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.575 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.839075_301.1", 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.839075159Z", 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.577 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.841110_302.1", 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36", 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.841110279Z", 2020-10-15T16:54:03.578 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:03.579 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.580 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.845049_303.1", 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-36", 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.845049758Z", 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.581 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.582 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.877008_304.1", 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:03.583 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.877008165Z", 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:03.584 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.881213_305.1", 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.881213458Z", 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.586 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.587 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.883257_306.1", 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37", 2020-10-15T16:54:03.588 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.883257547Z", 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.589 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.887331_307.1", 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-37", 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.887331707Z", 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.591 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.592 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.934865_308.1", 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.593 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.934865315Z", 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.594 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.939408_309.1", 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.939408181Z", 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.596 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.597 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.941448_310.1", 2020-10-15T16:54:03.598 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38", 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.941448054Z", 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780701.944929_311.1", 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-38", 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:41.944929850Z", 2020-10-15T16:54:03.601 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.668305_312.1", 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.668305083Z", 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.604 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.677868_314.1", 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:03.606 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.677868663Z", 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.607 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.682548_315.1", 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39", 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.682548949Z", 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.609 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.610 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.688701_316.1", 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.611 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-39", 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.688701616Z", 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.718879_317.1", 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.718879552Z", 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.614 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.724158_318.1", 2020-10-15T16:54:03.616 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.724158775Z", 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.618 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.726454_319.1", 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40", 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.726454595Z", 2020-10-15T16:54:03.619 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:03.620 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.621 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.731618_320.1", 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-40", 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.731618705Z", 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.622 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.623 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.794124_321.1", 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:03.624 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.794124321Z", 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:03.625 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.627 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.797914_322.1", 2020-10-15T16:54:03.627 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.627 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:03.627 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.797914867Z", 2020-10-15T16:54:03.627 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.627 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.628 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.630 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.800340_323.1", 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41", 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.800340181Z", 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:03.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780703.807546_324.1", 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-41", 2020-10-15T16:54:03.633 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.807546577Z", 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:03.634 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:03.635 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:03.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:03.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:03.641 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:03.681 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:03.682 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:03.917 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780022.078737_52.1", 2020-10-15T16:54:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "user", 2020-10-15T16:54:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.078737453Z", 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780022.087940_55.1", 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "user", 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw-multisite-test-user", 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:40:22.087940982Z", 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__8hgXc8dQ27EmK6yBHI0sRF", 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.020200_56.1", 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.020200486Z", 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.035923_61.1", 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1:803109df-3f66-4c18-857a-2beb852ab289.4346.1", 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.035923903Z", 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hE0-5bZwVLSmA8_GQvqpRj5", 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.037528_62.1", 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1", 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.037528840Z", 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.040794_63.1", 2020-10-15T16:54:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-1", 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.040794414Z", 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CJ2QVahv8uFTy6qtqmeOA1T", 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.063903_64.1", 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.063903600Z", 2020-10-15T16:54:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.068188_65.1", 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2:803109df-3f66-4c18-857a-2beb852ab289.4346.2", 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.068188141Z", 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VS6iqDndU2FgrR3oso3Kngk", 2020-10-15T16:54:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.070462_66.1", 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2", 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.070462424Z", 2020-10-15T16:54:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780145.074696_67.1", 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-2", 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:25.074696732Z", 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G0D0dRlHcnzrfWi5_C_bDJ5", 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.236399_68.1", 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.236399569Z", 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.246813_69.1", 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3:803109df-3f66-4c18-857a-2beb852ab289.4346.3", 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.246813559Z", 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_QB6Am1HbRVESmd4Y-D9FHSV", 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.249178_70.1", 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3", 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.249178308Z", 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780148.269888_71.1", 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-3", 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:28.269888245Z", 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_2iVZkavIdYbijtgPU7LmNLn", 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.562533_72.1", 2020-10-15T16:54:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.562533806Z", 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.569340_75.1", 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4:803109df-3f66-4c18-857a-2beb852ab289.4346.4", 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.569340542Z", 2020-10-15T16:54:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_jcyEhkV9rKaECF-o22iIqP3", 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.571388_76.1", 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4", 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.571388447Z", 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.578288_77.1", 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-4", 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.578288258Z", 2020-10-15T16:54:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_VHxIg1HjJ12aHd4HVq305SM", 2020-10-15T16:54:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.602337_78.1", 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.602337755Z", 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.607439_79.1", 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5:803109df-3f66-4c18-857a-2beb852ab289.4346.5", 2020-10-15T16:54:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.607439055Z", 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_gk-JJ1towyh-YcrRZR3t8Kx", 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.613564_80.1", 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5", 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.613564853Z", 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.619553_81.1", 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-5", 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.619553277Z", 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hnfmXHdhUxDyJ_-Bw8xlu_z", 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.677228_82.1", 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.677228758Z", 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.686417_83.1", 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6:803109df-3f66-4c18-857a-2beb852ab289.4346.6", 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.686417087Z", 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KKJKK12-8A6IhxX20xnDPJf", 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.691746_84.1", 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6", 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.691746539Z", 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780149.701238_85.1", 2020-10-15T16:54:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-6", 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:29.701238667Z", 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6K_c49VkkSKe3Rf-cIfkKNv", 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.610666_86.1", 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.610666330Z", 2020-10-15T16:54:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.615079_91.1", 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7:803109df-3f66-4c18-857a-2beb852ab289.4346.7", 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.615079770Z", 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_V2GjgIZhqahmxIuCIVYcjKo", 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.617109_92.1", 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.617109940Z", 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.624177_93.1", 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.624177067Z", 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.646587_94.1", 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.646587696Z", 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.651089_95.1", 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8:803109df-3f66-4c18-857a-2beb852ab289.4346.8", 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.651089989Z", 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Qy0IyT_uoqKD9mNKn5wEBSe", 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.653750_96.1", 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.653750502Z", 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.659158_97.1", 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.659158286Z", 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.005 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.006 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.705514_98.1", 2020-10-15T16:54:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.705514298Z", 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.714153_99.1", 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9:803109df-3f66-4c18-857a-2beb852ab289.4346.9", 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.714153152Z", 2020-10-15T16:54:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B2P4VBWKQiVIp45wzJGNz6m", 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.716741_100.1", 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.716741822Z", 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780152.720290_101.1", 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:32.720290421Z", 2020-10-15T16:54:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780153.973023_102.1", 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:33.973023863Z", 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "remove" 2020-10-15T16:54:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780153.980103_105.1", 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-7", 2020-10-15T16:54:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:33.980103075Z", 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_nn8HQr8mZtAvIUhlEuk5cO9", 2020-10-15T16:54:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.037669_106.1", 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.037669362Z", 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "remove" 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.044552_107.1", 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-8", 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.044552548Z", 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0JxLFcQ8AiR4Kpr39W5tF5d", 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.151609_108.1", 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.151609693Z", 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.028 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.029 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "remove" 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780154.156484_109.1", 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.030 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-9", 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484486Z", 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.031 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6PultVJ3L2RFVNn3HFdHTHP", 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.035 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.436554_75.1", 2020-10-15T16:54:04.035 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.035 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.436554576Z", 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.036 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.441203_78.1", 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10", 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.441203519Z", 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.038 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_rVYe4R9j6IeSXdNvRG5czSr", 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.039 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.444010_79.1", 2020-10-15T16:54:04.040 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10", 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.444010714Z", 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.041 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.448982_80.1", 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-10", 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.448982102Z", 2020-10-15T16:54:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_w1Y1QsEGmarrzsetwPC7iZM", 2020-10-15T16:54:04.044 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.045 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.471604_81.1", 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.471604415Z", 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.046 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.047 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.476178_82.1", 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.048 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11", 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.476178067Z", 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.049 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_i8qP9olAuVVWxXhiPMeOKL-", 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.478035_83.1", 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11", 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.478035273Z", 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.051 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.052 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.483334_84.1", 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-11", 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.483334680Z", 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.054 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ED7eVH0kLvHPgegocuwR32u", 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.536455_85.1", 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.536455959Z", 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.057 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.547313_86.1", 2020-10-15T16:54:04.058 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12", 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.547313800Z", 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BCF9r_IExK2bDBXai46ANvH", 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.549635_87.1", 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12", 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.549635460Z", 2020-10-15T16:54:04.061 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.063 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780155.557492_88.1", 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-12", 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:35.557492671Z", 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_bmEuUGKkp4IyDTTGJzFlfYF", 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.065 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.661175_92.1", 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:04.066 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.661175781Z", 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:04.067 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.665404_96.1", 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13", 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.665404097Z", 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.069 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RxDkInfwxXqU-foe6BOIrg6", 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.070 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.667489_97.1", 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13", 2020-10-15T16:54:04.071 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.667489966Z", 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.671455_98.1", 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-13", 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.671455049Z", 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_sRwunfB8lpSMnPcgAQ2Rmy1", 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.075 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.693318_99.1", 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.076 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.693318799Z", 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.697718_100.1", 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14", 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.697718647Z", 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NmE8zGBvHYMxZX3QlJ3KBXF", 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.700284_101.1", 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.081 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14", 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.700284160Z", 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.082 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.704804_102.1", 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-14", 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.704804291Z", 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.084 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_x2zTJ66iStP1cOplhi-J9Q-", 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.086 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.750755_103.1", 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.750755269Z", 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.759406_104.1", 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15", 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.759406919Z", 2020-10-15T16:54:04.089 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_-LnuMiImi2Uw_3WIjT6-mit", 2020-10-15T16:54:04.090 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.761217_105.1", 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15", 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.761217640Z", 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.092 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.093 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780166.765834_106.1", 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.094 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-15", 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:46.765834376Z", 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_u9dTqV9Q3B0QoLpimNzJiy4", 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.955052_107.1", 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.955052062Z", 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.097 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.098 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.959162_113.1", 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.099 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.959162975Z", 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.100 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.961086_114.1", 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16", 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.961086719Z", 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.102 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.965760_115.1", 2020-10-15T16:54:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16", 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.965760332Z", 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_FshGfcSuZ1Fq0NPuhBDrUzY", 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.992057_116.1", 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.992057998Z", 2020-10-15T16:54:04.107 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:04.108 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780215.998223_117.1", 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:35.998223727Z", 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.111 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.000912_118.1", 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17", 2020-10-15T16:54:04.112 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.000912498Z", 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:04.113 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.005187_119.1", 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17", 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.005187083Z", 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.115 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_GZltXLwr7I44hl3cdjzca_y", 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.116 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.060953_120.1", 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.117 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.060953979Z", 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.118 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.067276_121.1", 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.067276061Z", 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.120 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.121 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.069062_122.1", 2020-10-15T16:54:04.122 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18", 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.069062558Z", 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.123 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.072845_123.1", 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18", 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.072845282Z", 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.125 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qY9cdhGNg-4mOAG71J3vMam", 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.127 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.093654_124.1", 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.093654456Z", 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.099372_125.1", 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16", 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.099372812Z", 2020-10-15T16:54:04.130 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_RXs8n4w_f7FqXrzzbGE0Bms", 2020-10-15T16:54:04.131 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.104083_126.1", 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.104083201Z", 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.134 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.110041_127.1", 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17", 2020-10-15T16:54:04.135 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.110041428Z", 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__lucWH-upnNjAcmQvU8boak", 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.123317_128.1", 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.123317028Z", 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780216.136755_129.1", 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18", 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:43:36.136755384Z", 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tHkvXjVV_bl-ExJET1sAXYR", 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.237117_130.1", 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.237117211Z", 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.242376_146.1", 2020-10-15T16:54:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.242376619Z", 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.244152_147.1", 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19", 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.244152088Z", 2020-10-15T16:54:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.249725_148.1", 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19", 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.249725487Z", 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_J7t9-GONbjRRUIyXTXBzdyC", 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.258102_149.1", 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.258102372Z", 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780409.263968_150.1", 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19", 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:49.263968538Z", 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yb-0YCtPspR6kndJAqPSveY", 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.774083_151.1", 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.774083854Z", 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.778075_162.1", 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.778075584Z", 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.779994_163.1", 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20", 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.779994982Z", 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780568.783569_164.1", 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20", 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:28.783569181Z", 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_04naSkvluQtUIchxhrL5xEU", 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780573.133653_165.1", 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:33.133653795Z", 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780573.141127_169.1", 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:49:33.141127542Z", 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780638.077273_184.1", 2020-10-15T16:54:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:50:38.077273627Z", 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780638.085516_186.1", 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20", 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:50:38.085516566Z", 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_7EWutVJtN4j4Kx6bs16S-q1", 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.950817_187.1", 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.950817471Z", 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.955895_191.1", 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.955895323Z", 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.957899_192.1", 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21", 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.957899082Z", 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.963694_193.1", 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21", 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.963694041Z", 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_kBrZ1Q7CarMPezPOOxJSre5", 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.989141_194.1", 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.989141836Z", 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.993921_195.1", 2020-10-15T16:54:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.993921692Z", 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780677.996598_196.1", 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22", 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:17.996598844Z", 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.001078_197.1", 2020-10-15T16:54:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22", 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.001078013Z", 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__0-tVUail2LR0-_dvdtyTmN", 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.046900_198.1", 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.046900555Z", 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.051294_199.1", 2020-10-15T16:54:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.051294115Z", 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.053404_200.1", 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23", 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.053404083Z", 2020-10-15T16:54:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.057559_201.1", 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23", 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.057559943Z", 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_JgFzUrsGSOjHpDuuQvOIs5a", 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.078368_202.1", 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.078368231Z", 2020-10-15T16:54:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.083939_203.1", 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21", 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.083939714Z", 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BqmH40YtrMs3oXCzZvWv4lq", 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.088683_204.1", 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.088683934Z", 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.094969_205.1", 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22", 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.094969549Z", 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NntuH58QnIACtCjzKai1wEE", 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.106109_206.1", 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.106109849Z", 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780678.113514_207.1", 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23", 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:18.113514114Z", 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PmqDQrt62VvZTyWLiKlRk4S", 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.522956_208.1", 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.522956427Z", 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.526974_216.1", 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.526974067Z", 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.529080_217.1", 2020-10-15T16:54:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24", 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.529080378Z", 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.532761_218.1", 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24", 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.532761600Z", 2020-10-15T16:54:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_iKpTfoHIWZn2VQK4CdoAv1d", 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.558298_219.1", 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.558298155Z", 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.562423_220.1", 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.562423620Z", 2020-10-15T16:54:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.564100_221.1", 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25", 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.564100877Z", 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.567945_222.1", 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25", 2020-10-15T16:54:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.567945059Z", 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_KEyWZxh5Pk-ZuI7R87WiGxN", 2020-10-15T16:54:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.612199_223.1", 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.612199960Z", 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.616295_224.1", 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.616295813Z", 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.618054_225.1", 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26", 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.618054466Z", 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.622228_226.1", 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26", 2020-10-15T16:54:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.622228203Z", 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_P8Jz4lg4H-B5bwCjnZ6fRNr", 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.640971_227.1", 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.640971196Z", 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.650996_228.1", 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.257 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.650996655Z", 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.258 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.654993_229.1", 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.654993267Z", 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.260 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.261 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.663850_230.1", 2020-10-15T16:54:04.262 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24", 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.663850522Z", 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.263 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UDchd-hJg5yhLUsYvAHfN7e", 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.670414_231.1", 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.670414149Z", 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.265 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.677960_232.1", 2020-10-15T16:54:04.267 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.677960563Z", 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.689991_233.1", 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.689991599Z", 2020-10-15T16:54:04.270 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.272 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.699262_234.1", 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25", 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.699262282Z", 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hQAUpC2r8Q9PXJ7lmyLO153", 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.717126_235.1", 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.717126422Z", 2020-10-15T16:54:04.275 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.277 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.726779_236.1", 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.726779903Z", 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.278 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.739244_237.1", 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:04.280 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.739244614Z", 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.281 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780688.747532_238.1", 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26", 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:28.747532325Z", 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.283 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4YhF2CxEaqoiMI2n4QnieBd", 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 3 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.725458_239.1", 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:04.285 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.725458610Z", 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:04.286 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.729952_247.1", 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.729952098Z", 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.288 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.289 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.732194_248.1", 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27", 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.732194555Z", 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.736099_249.1", 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27", 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.736099973Z", 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.293 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0U_xBRG1MxH3kTiTCOLJrTA", 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.294 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.763018_250.1", 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.763018073Z", 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.767193_251.1", 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.767193540Z", 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.299 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.769081_252.1", 2020-10-15T16:54:04.300 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28", 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.769081040Z", 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.301 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.772622_253.1", 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28", 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.772622294Z", 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.303 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_psT16E79fNh1QdYexVGx4o7", 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.820914_254.1", 2020-10-15T16:54:04.305 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.820914739Z", 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.307 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.828708_255.1", 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.828708805Z", 2020-10-15T16:54:04.308 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:04.309 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.310 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.831143_256.1", 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29", 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.831143283Z", 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.836443_257.1", 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29", 2020-10-15T16:54:04.313 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.836443295Z", 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CIQuhrBK2-xtlyniK0TojzF", 2020-10-15T16:54:04.314 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.861145_258.1", 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.861145621Z", 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:04.316 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.317 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.867657_259.1", 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-27:803109df-3f66-4c18-857a-2beb852ab289.4346.27", 2020-10-15T16:54:04.318 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.867657187Z", 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.319 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "__KdZlC6AOwtkiOKCjaIyf5b", 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.916588_260.1", 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.916588779Z", 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.322 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.927095_261.1", 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-28:803109df-3f66-4c18-857a-2beb852ab289.4346.28", 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.927095157Z", 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.324 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5Hn16rsVCsUD-bwB-j_SgpP", 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.987203_262.1", 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.987203501Z", 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.326 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.327 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780698.991971_263.1", 2020-10-15T16:54:04.328 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-29:803109df-3f66-4c18-857a-2beb852ab289.4346.29", 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:38.991971647Z", 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.329 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_aiZ2fzgk2l6QBfsl7HNRyfu", 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.072759_264.1", 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.072759848Z", 2020-10-15T16:54:04.331 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.080572_265.1", 2020-10-15T16:54:04.333 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.080572799Z", 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.083324_266.1", 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30", 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.083324055Z", 2020-10-15T16:54:04.336 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.338 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.087985_267.1", 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30", 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.087985505Z", 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.339 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_6eLQUuX5gjNvcgGHSucZ_Cr", 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.115281_268.1", 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:04.341 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.115281942Z", 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:04.342 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.119612_269.1", 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.119612435Z", 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.344 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.122031_270.1", 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31", 2020-10-15T16:54:04.346 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.122031946Z", 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.126491_271.1", 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31", 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.126491865Z", 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_ESI-XZb0gNA7y8OwabjPxKS", 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.350 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.170827_272.1", 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:04.351 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.170827362Z", 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.352 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.177992_273.1", 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.177992630Z", 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.354 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.355 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.188242_274.1", 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.356 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32", 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.188242974Z", 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.357 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.193410_275.1", 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32", 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.193410108Z", 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.359 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_CM5E1AFKgN9ls1gqt9nHnam", 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.360 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.228520_276.1", 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.361 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.228520778Z", 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.362 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.233934_277.1", 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-30:803109df-3f66-4c18-857a-2beb852ab289.4346.30", 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.233934127Z", 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.364 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_DOUrz1JwTAzHmr8tDZoEJze", 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.365 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.285734_278.1", 2020-10-15T16:54:04.366 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.285734573Z", 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.367 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.295813_279.1", 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-31:803109df-3f66-4c18-857a-2beb852ab289.4346.31", 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.295813271Z", 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.369 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PIcF2Cy8wcVcllEpXGYZ0ls", 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.358209_280.1", 2020-10-15T16:54:04.371 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.358209390Z", 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.372 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.369099_281.1", 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-32:803109df-3f66-4c18-857a-2beb852ab289.4346.32", 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.369099338Z", 2020-10-15T16:54:04.374 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_B_T4vHpUEXSBWDDiLO8hMZP", 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.440343_282.1", 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.440343394Z", 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.444859_283.1", 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.444859665Z", 2020-10-15T16:54:04.379 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:04.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.381 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.446812_284.1", 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33", 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.446812694Z", 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.451294_285.1", 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33", 2020-10-15T16:54:04.384 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.451294634Z", 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_NLby68w8cYPIluW5Adnq7QA", 2020-10-15T16:54:04.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.478516_286.1", 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.478516894Z", 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.388 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.483226_287.1", 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.483226132Z", 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.485601_288.1", 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34", 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.485601115Z", 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.492457_289.1", 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34", 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.492457380Z", 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_G9-ZgYWayYpE_p72eu-RvJ5", 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.553617_290.1", 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.553617175Z", 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.563843_291.1", 2020-10-15T16:54:04.399 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.563843665Z", 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.400 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.568864_292.1", 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35", 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.568864246Z", 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.402 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.403 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.584697_293.1", 2020-10-15T16:54:04.404 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35", 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.584697168Z", 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.405 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_quNCUuTVUR19_S4LWmzkTWF", 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.607874_294.1", 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.607874022Z", 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.407 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780699.624681_295.1", 2020-10-15T16:54:04.409 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-33:803109df-3f66-4c18-857a-2beb852ab289.4346.33", 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:39.624681194Z", 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qn6PECb8-hvVq0RT1SicuPv", 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780700.672575_296.1", 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:40.672575058Z", 2020-10-15T16:54:04.412 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:04.413 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.414 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780700.684269_297.1", 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-34:803109df-3f66-4c18-857a-2beb852ab289.4346.34", 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:40.684269312Z", 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:04.415 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_LmA9l4ETi1bujEjdo03q-tx", 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.416 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.744760_298.1", 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:04.417 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.744760968Z", 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.758273_299.1", 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-35:803109df-3f66-4c18-857a-2beb852ab289.4346.35", 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.758273297Z", 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.420 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_4gwSphs1xbSxWUtWEVn7pZX", 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 2 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.421 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.834807_300.1", 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.422 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.834807066Z", 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.839075_301.1", 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36", 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.839075159Z", 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_PoivHo81Ml19vdPNnuL0-_t", 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.426 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.841110_302.1", 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36", 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.841110279Z", 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.428 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.845049_303.1", 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-36", 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.845049758Z", 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.430 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tYlMrQ1U73LhmTftJQqeTbU", 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.431 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.877008_304.1", 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.432 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.877008165Z", 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.433 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.881213_305.1", 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37", 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.881213458Z", 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.435 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_UEgO55vo7MbtGs59cSiuj_f", 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.436 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.883257_306.1", 2020-10-15T16:54:04.437 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37", 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.883257547Z", 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.438 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.887331_307.1", 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-37", 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.887331707Z", 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.440 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_qDrySzqAHaqPCGQPT4wflG8", 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.934865_308.1", 2020-10-15T16:54:04.442 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.934865315Z", 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.939408_309.1", 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38:803109df-3f66-4c18-857a-2beb852ab289.4346.38", 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.939408181Z", 2020-10-15T16:54:04.445 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_hHUj4TsjqpkdmXPCpqEYVUH", 2020-10-15T16:54:04.446 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.447 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.941448_310.1", 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38", 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.941448054Z", 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.448 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.449 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780701.944929_311.1", 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-38", 2020-10-15T16:54:04.450 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:41.944929850Z", 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_tzMu3BZ4cCVBaGxb7Va2AQU", 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.668305_312.1", 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.668305083Z", 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.454 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.677868_314.1", 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39:803109df-3f66-4c18-857a-2beb852ab289.4346.39", 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.677868663Z", 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.456 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_djqbQlOxq_ZvjJoKRn2lQD2", 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.682548_315.1", 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39", 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.682548949Z", 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.459 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.688701_316.1", 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.460 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-39", 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.688701616Z", 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.461 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_l9BlYI5-K0J6nchGdDtQHVS", 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.718879_317.1", 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.718879552Z", 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.463 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.464 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.724158_318.1", 2020-10-15T16:54:04.465 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40:803109df-3f66-4c18-857a-2beb852ab289.4346.40", 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.724158775Z", 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_O8r3wd-4Qe1-vlSBYxQGbFU", 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.726454_319.1", 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40", 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.726454595Z", 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.468 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.731618_320.1", 2020-10-15T16:54:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-40", 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.731618705Z", 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_zzenoG5PtlSoJiQlMgHOqRz", 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.794124_321.1", 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.794124321Z", 2020-10-15T16:54:04.473 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.475 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.797914_322.1", 2020-10-15T16:54:04.476 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.476 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41:803109df-3f66-4c18-857a-2beb852ab289.4346.41", 2020-10-15T16:54:04.476 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.797914867Z", 2020-10-15T16:54:04.476 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.476 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.477 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.477 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_HH8l4tOW_MNJM1mPzSOl12f", 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.478 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.800340_323.1", 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41", 2020-10-15T16:54:04.479 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.800340181Z", 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780703.807546_324.1", 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-41", 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.807546577Z", 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:04.482 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_eS5MazU4AXh3Bge7-B-32-7", 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:04.483 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:04.484 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:04.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:04.484 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:04.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:04.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:04.486 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:04.524 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:04.525 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:04.747 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:04.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.747 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.831026_168.1", 2020-10-15T16:54:04.747 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.747 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:04.748 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.831026973Z", 2020-10-15T16:54:04.748 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.748 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.748 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.748 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.749 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.750 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.835741_169.1", 2020-10-15T16:54:04.750 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.750 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:04.750 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.835741715Z", 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:04.751 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.838500_170.1", 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42", 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.838500484Z", 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.753 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.754 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.844130_171.1", 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:04.755 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42", 2020-10-15T16:54:04.756 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.844130996Z", 2020-10-15T16:54:04.756 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.756 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.756 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.756 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.757 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.757 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:04.757 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.757 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.757 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.894539_172.1", 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.758 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.894539034Z", 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.759 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.899113_173.1", 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.899113027Z", 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.761 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.763 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.901152_174.1", 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43", 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.901152073Z", 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.765 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.765 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:04.765 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.765 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.765 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:04.766 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.766 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:04.766 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.905271_175.1", 2020-10-15T16:54:04.766 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:04.767 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43", 2020-10-15T16:54:04.767 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.905271712Z", 2020-10-15T16:54:04.767 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:04.767 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:04.767 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:04.768 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:04.776 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:04.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:04.777 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:04.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:04.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:04.779 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:04.820 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:04.820 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:05.063 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:05.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.064 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.831026_168.1", 2020-10-15T16:54:05.064 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.064 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:05.065 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.831026973Z", 2020-10-15T16:54:05.065 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.065 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.065 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.065 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.066 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.067 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:05.067 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.067 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.067 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.068 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.835741_169.1", 2020-10-15T16:54:05.068 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.835741715Z", 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.070 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:05.070 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.070 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.070 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.838500_170.1", 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.071 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42", 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.838500484Z", 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.074 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.844130_171.1", 2020-10-15T16:54:05.074 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.074 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42", 2020-10-15T16:54:05.074 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.844130996Z", 2020-10-15T16:54:05.074 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.075 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.075 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.075 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.075 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.076 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:05.076 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.076 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.076 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.077 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.894539_172.1", 2020-10-15T16:54:05.077 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.894539034Z", 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.079 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:05.079 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.079 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.899113_173.1", 2020-10-15T16:54:05.080 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.899113027Z", 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.901152_174.1", 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43", 2020-10-15T16:54:05.083 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.901152073Z", 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.086 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.905271_175.1", 2020-10-15T16:54:05.086 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.086 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43", 2020-10-15T16:54:05.086 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.905271712Z", 2020-10-15T16:54:05.086 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.086 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.088 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.098 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.099 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:05.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:05.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:05.101 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:05.140 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:05.141 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:05.360 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:05.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.360 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.831026_168.1", 2020-10-15T16:54:05.361 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.361 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:05.361 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.831026973Z", 2020-10-15T16:54:05.361 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.361 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.361 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.362 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.835741_169.1", 2020-10-15T16:54:05.363 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.835741715Z", 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:05.365 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.838500_170.1", 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42", 2020-10-15T16:54:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.838500484Z", 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:05.367 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.844130_171.1", 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-42", 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.844130996Z", 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.369 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.370 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.894539_172.1", 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.371 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.894539034Z", 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.372 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.899113_173.1", 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.899113027Z", 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.374 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.375 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.901152_174.1", 2020-10-15T16:54:05.376 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43", 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.901152073Z", 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.378 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780771.905271_175.1", 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-43", 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:52:51.905271712Z", 2020-10-15T16:54:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.381 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.394 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:05.395 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.395 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.395 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.395 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:05.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:05.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:05.398 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:05.436 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:05.436 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:05.661 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:05.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.662 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.831026_168.1", 2020-10-15T16:54:05.662 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.662 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:05.662 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.831026973Z", 2020-10-15T16:54:05.662 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.662 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.664 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.835741_169.1", 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1", 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.835741715Z", 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.666 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_5CCaGji2BsCMecl2FhZHNd8", 2020-10-15T16:54:05.666 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.666 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.666 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.838500_170.1", 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42", 2020-10-15T16:54:05.667 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.838500484Z", 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.844130_171.1", 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-42", 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.844130996Z", 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.670 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_cNY9j5juVF8gCc_hERIJysU", 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.671 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.894539_172.1", 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.672 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.894539034Z", 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.673 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.899113_173.1", 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43:669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2", 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.899113027Z", 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.675 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_BLCLUJu97C3E3WHesayD1FW", 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.677 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.901152_174.1", 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43", 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.901152073Z", 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:05.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780771.905271_175.1", 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-43", 2020-10-15T16:54:05.680 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:52:51.905271712Z", 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_yMe2_iVsoggNDpyRxc2ubyw", 2020-10-15T16:54:05.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:05.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:05.682 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:05.694 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:05.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:05.695 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:05.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:05.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:05.699 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:05.739 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:05.740 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:05.987 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:05.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.988 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.074256_220.1", 2020-10-15T16:54:05.988 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.988 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:05.988 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.074256469Z", 2020-10-15T16:54:05.988 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.988 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.989 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.990 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:05.990 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.990 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.990 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.079388_221.1", 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.079388553Z", 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:05.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.082036_222.1", 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44", 2020-10-15T16:54:05.993 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.082036256Z", 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:05.994 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.087874_223.1", 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44", 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.087874321Z", 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.996 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:05.997 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.131149_224.1", 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:05.998 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.131149120Z", 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:05.999 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.136636_225.1", 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.136636156Z", 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.001 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.138948_226.1", 2020-10-15T16:54:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.138948841Z", 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:06.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.143078_227.1", 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.143078013Z", 2020-10-15T16:54:06.006 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.007 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.008 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.022 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:06.022 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.022 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.022 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.023 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:06.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:06.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:06.025 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:06.064 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:06.064 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:06.295 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:06.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.296 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.074256_220.1", 2020-10-15T16:54:06.296 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.074256469Z", 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.297 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.079388_221.1", 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.079388553Z", 2020-10-15T16:54:06.299 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.301 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.082036_222.1", 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44", 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.082036256Z", 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.302 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.087874_223.1", 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44", 2020-10-15T16:54:06.304 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.087874321Z", 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:06.306 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.306 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.306 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.307 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.131149_224.1", 2020-10-15T16:54:06.307 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.307 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.131149120Z", 2020-10-15T16:54:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.308 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.136636_225.1", 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.136636156Z", 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.310 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.312 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.138948_226.1", 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.138948841Z", 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.143078_227.1", 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.315 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.143078013Z", 2020-10-15T16:54:06.316 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.316 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.316 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:06.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:06.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:06.333 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:06.371 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:06.372 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:06.596 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.074256_220.1", 2020-10-15T16:54:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.074256469Z", 2020-10-15T16:54:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.079388_221.1", 2020-10-15T16:54:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.079388553Z", 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.082036_222.1", 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44", 2020-10-15T16:54:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.082036256Z", 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.087874_223.1", 2020-10-15T16:54:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-44", 2020-10-15T16:54:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.087874321Z", 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.131149_224.1", 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.131149120Z", 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.136636_225.1", 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.136636156Z", 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.138948_226.1", 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.138948841Z", 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "write" 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "1_1602780833.143078_227.1", 2020-10-15T16:54:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "section": "bucket", 2020-10-15T16:54:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:53:53.143078013Z", 2020-10-15T16:54:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "data": { 2020-10-15T16:54:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "read_version": { 2020-10-15T16:54:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "", 2020-10-15T16:54:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 0 2020-10-15T16:54:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "write_version": { 2020-10-15T16:54:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "ver": 1 2020-10-15T16:54:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "status": { 2020-10-15T16:54:06.629 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "complete" 2020-10-15T16:54:06.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:06.630 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:06.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:06.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:06.633 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:06.672 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:06.672 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:06.911 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:06.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.912 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.074256_220.1", 2020-10-15T16:54:06.912 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.912 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:06.912 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.074256469Z", 2020-10-15T16:54:06.912 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.079388_221.1", 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44:803109df-3f66-4c18-857a-2beb852ab289.5150.1", 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.079388553Z", 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.915 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_0gftkYF-PJRrpZ1nwNGosTe", 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.916 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.082036_222.1", 2020-10-15T16:54:06.917 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44", 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.082036256Z", 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.087874_223.1", 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-44", 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.087874321Z", 2020-10-15T16:54:06.920 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_H_BTVZmwDppJXyFCqL06TiR", 2020-10-15T16:54:06.921 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.922 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.131149_224.1", 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.131149120Z", 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.136636_225.1", 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket.instance", 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45:803109df-3f66-4c18-857a-2beb852ab289.5150.2", 2020-10-15T16:54:06.925 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.136636156Z", 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.926 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_Zrp1I9NhcGfWH8QZNrmfA4u", 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.138948_226.1", 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.138948841Z", 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.928 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "write" 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "1_1602780833.143078_227.1", 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: "section": "bucket", 2020-10-15T16:54:06.930 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iybjjr-45", 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:53:53.143078013Z", 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: "data": { 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: "read_version": { 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "", 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 0 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.931 INFO:teuthology.orchestra.run.smithi038.stdout: "write_version": { 2020-10-15T16:54:06.932 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "_9q6iljkacCdiUMsz4cPOlct", 2020-10-15T16:54:06.932 INFO:teuthology.orchestra.run.smithi038.stdout: "ver": 1 2020-10-15T16:54:06.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:06.932 INFO:teuthology.orchestra.run.smithi038.stdout: "status": { 2020-10-15T16:54:06.943 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "complete" 2020-10-15T16:54:06.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:06.944 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:06.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:06.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:06.946 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:06.982 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:06.982 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:07.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:07.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:07.297 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:07.339 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:07.339 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:07.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:07.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:07.775 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:07.811 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:07.811 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:08.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:08.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:08.072 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:08.109 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:08.109 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:08.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:08.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:08.492 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:08.529 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:08.529 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:08.798 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:54:08.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:08.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:08.801 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:08.841 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:08.842 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:09.108 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:54:09.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:09.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:09.111 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:09.150 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:09.150 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:09.403 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:54:09.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:09.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0c649eb8-3e99-4d22-800d-adf2e2823b92', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:09.406 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0c649eb8-3e99-4d22-800d-adf2e2823b92 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:09.724 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:54:09.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:09.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:09.727 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:09.765 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:09.765 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:10.020 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:54:10.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.023 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:10.062 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:10.062 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:10.317 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:54:10.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.320 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:10.359 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:10.359 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:10.621 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:54:10.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b84643c0-8302-4cfb-a70b-7af9afa8e893', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.624 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b84643c0-8302-4cfb-a70b-7af9afa8e893 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:10.663 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:10.664 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:10.936 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:54:10.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:10.939 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:10.978 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:10.978 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:11.211 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:54:11.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:11.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:11.214 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:11.249 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:11.249 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:11.532 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:54:11.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:11.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:11.536 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:11.575 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:11.575 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:11.843 INFO:teuthology.orchestra.run.smithi183.stdout:[] 2020-10-15T16:54:11.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:11.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '69fb12ff-ae85-466c-bb2d-3018777b1732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:11.847 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 69fb12ff-ae85-466c-bb2d-3018777b1732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:11.888 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:11.889 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:12.152 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2020-10-15T16:54:12.153 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-10-15T16:54:12.154 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-46 2020-10-15T16:54:12.193 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-47 2020-10-15T16:54:12.269 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-48 2020-10-15T16:54:12.355 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:54:12.348+0000 7f56957fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T16:54:12.356 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T16:54:12.351+0000 7ffb6bff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T16:54:12.357 INFO:tasks.rgw.c2.client.0.smithi038.stdout:2020-10-15T16:54:12.355+0000 7ff0faffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T16:54:12.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:12.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:12.483 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:12.522 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:12.522 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:12.744 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T16:54:12.773 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:12.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:12.774 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:54:12.774 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:54:12.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:12.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:12.774 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:12.307429Z" 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:12.775 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:54:12.776 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:54:12.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:12.776 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:12.777 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:54:12.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:12.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:12.778 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:12.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:12.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:13.066 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:13.066 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:54:13.066 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:54:13.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:54:13.068 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:54:13.069 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:54:13.070 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.071 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:54:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:54:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:54:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.075 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:13.076 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:54:13.076 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:54:13.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.077 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:13.117 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:13.117 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:13.354 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:54:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:54:13.354 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:54:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:54:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:54:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:54:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:54:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.355 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:54:13.356 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:54:13.357 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:13.358 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:54:13.358 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:54:13.358 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:13.358 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:13.358 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:13.358 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:13.359 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:54:13.360 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:54:13.361 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:13.362 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:13.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:13.362 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:54:13.362 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:54:13.362 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.362 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.363 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:54:13.364 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:54:13.365 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:54:13.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.366 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:13.406 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:13.407 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:13.665 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:13.665 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:54:13.666 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:54:13.666 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:54:13.666 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:54:13.666 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:54:13.667 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:54:13.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.667 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:54:13.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.667 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.668 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:54:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:54:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:54:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:54:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.670 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:54:13.671 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:54:13.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.671 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.672 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:13.673 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:54:13.674 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:54:13.674 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:13.674 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:13.674 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:13.674 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:54:13.675 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:54:13.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:13.676 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:13.677 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:54:13.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.678 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:13.717 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:13.717 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:13.961 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:13.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.961 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:13.962 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:13.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.962 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:13.962 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:13.963 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:13.964 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:13.964 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:13.964 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:13.965 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-15T16:54:13.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:13.966 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:14.008 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:14.008 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:14.252 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:14.252 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 2547530715017092207 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:54:14.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:14.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:54:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:14.260 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:14.261 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-15T16:54:14.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:14.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:14.263 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:14.302 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:14.303 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:14.573 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:14.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.574 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:14.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:14.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.575 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:14.575 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:14.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.576 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:14.576 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:14.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.577 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:14.578 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:14.578 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.578 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:14.579 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-15T16:54:14.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:14.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:14.582 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:14.662 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:14.663 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:14.922 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:54:14.923 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:54:14.923 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:54:14.923 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:54:14.924 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:54:14.924 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 1541412607139480694 2020-10-15T16:54:14.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.925 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:54:14.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.925 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:54:14.925 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:14.926 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.926 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:14.926 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:14.926 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:54:14.927 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:14.928 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:14.929 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:14.930 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:14.931 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:14.932 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:54:14.933 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-15T16:54:14.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:14.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:14.935 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:15.015 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:15.016 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:15.257 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:15.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:15.258 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:15.258 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:15.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:15.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:15.258 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:15.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:15.260 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:15.260 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:15.260 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:15.260 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:15.261 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-15T16:54:15.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:15.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:15.262 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:15.673 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:15.674 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:54:15.674 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:54:15.674 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:54:15.674 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:54:15.674 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 16587652486901249284 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:15.677 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:54:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:15.680 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:15.681 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:54:15.682 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T16:54:15.682 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T16:54:15.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:15.682 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:15.683 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-15T16:54:15.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:15.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:15.685 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:15.762 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:16.042 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-15T16:54:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:54:13.624776Z" 2020-10-15T16:54:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.043 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:54:16.044 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:54:16.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:16.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.044 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:16.045 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:51:43.561184Z" 2020-10-15T16:54:16.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:16.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.046 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:54:16.046 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:54:16.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:16.046 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:16.047 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-15T16:54:16.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.049 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:16.092 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:16.093 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:16.349 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:54:16.349 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:54:16.350 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:54:16.350 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:54:16.350 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:54:16.351 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 6408475988550126921 2020-10-15T16:54:16.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.351 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:54:16.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.352 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:54:16.352 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.352 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.353 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-15T16:54:16.353 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.353 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.354 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.354 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:13.624776Z" 2020-10-15T16:54:16.354 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.355 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:54:16.355 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.356 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.356 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:54:16.356 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.356 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.357 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.357 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:54:16.357 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.358 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:54:16.358 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.359 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.359 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:16.359 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.360 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.360 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.360 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.561184Z" 2020-10-15T16:54:16.360 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.361 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:54:16.361 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.362 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.362 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:54:16.362 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.363 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.363 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.363 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:54:16.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.364 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.364 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:54:16.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.365 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:54:16.365 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:54:16.365 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:54:16.365 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.366 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:54:16.367 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-15T16:54:16.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.369 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:16.411 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:16.412 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:16.679 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-15T16:54:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:54:13.624776Z" 2020-10-15T16:54:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:54:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:54:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:51:43.561184Z" 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:16.682 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:16.683 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-15T16:54:16.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.685 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:16.727 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:16.727 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:16.971 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:54:16.972 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:54:16.972 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:54:16.972 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:54:16.972 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:54:16.972 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 5492724315337703495 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.973 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:13.624776Z" 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.975 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.976 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.561184Z" 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:16.978 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:54:16.979 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:54:16.980 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:54:16.980 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:16.980 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:54:16.981 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-15T16:54:16.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:16.982 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:17.072 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:17.072 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:17.313 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:17.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.313 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:12.610216Z" 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.654924Z" 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:54:17.315 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.316 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:17.316 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-15T16:54:17.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:17.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:17.318 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:17.402 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:17.402 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:17.665 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:54:17.665 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:54:17.665 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:54:17.665 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 4508126650953487876 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:54:17.666 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.610216Z" 2020-10-15T16:54:17.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:54:17.668 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.669 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:51:43.654924Z" 2020-10-15T16:54:17.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:17.671 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:54:17.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:17.673 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:54:17.673 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T16:54:17.673 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T16:54:17.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:17.673 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:54:17.674 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-15T16:54:17.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:17.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:17.675 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:17.753 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:17.753 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:17.999 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:17.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:17.999 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-15T16:54:17.999 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:12.610216Z" 2020-10-15T16:54:18.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:18.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.000 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:54:18.000 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:54:18.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:18.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.001 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:18.001 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.654924Z" 2020-10-15T16:54:18.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:18.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.001 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:18.001 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:54:18.002 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:18.002 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:18.003 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-15T16:54:18.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:18.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:18.004 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:18.047 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:18.047 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:18.288 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:18.288 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 11637474988163193673 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:54:18.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:18.290 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:54:18.290 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:18.290 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:18.290 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-15T16:54:18.290 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:18.290 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.610216Z" 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:54:18.291 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:47:34.675980Z" 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:18.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:18.293 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.654924Z" 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:54:18.294 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.173072Z" 2020-10-15T16:54:18.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:18.296 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:18.297 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-15T16:54:18.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:18.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:18.299 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:18.335 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:18.336 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:18.591 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:18.594 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:18.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:18.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:18.596 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:18.672 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:18.672 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:18.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:18.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:18.938 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:18.975 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:18.975 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:19.264 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T16:54:19.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:54:13.624776Z" 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005433", 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:47:34.654913Z" 2020-10-15T16:54:19.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:51:43.561184Z" 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-15T16:54:19.266 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T16:46:07.994158Z" 2020-10-15T16:54:19.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:19.267 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T16:54:19.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:19.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:19.269 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:19.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:19.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:54:19.634 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:54:19.675 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:19.675 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:19.934 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:54:13.436770Z" 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006098", 2020-10-15T16:54:19.935 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:47:34.738031Z" 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:51:43.664298Z" 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:54:19.936 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T16:54:19.937 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:46:08.169845Z" 2020-10-15T16:54:19.937 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:54:19.937 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:54:19.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:19.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:19.939 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:20.017 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:20.018 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:20.279 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-10-15T16:54:20.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:20.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:20.281 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:54:20.350+0000 7fc8937000c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:54:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:54:20.350+0000 7fc8937000c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:54:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:54:20.350+0000 7fc8937000c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:54:20.355 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:54:20.350+0000 7fc8937000c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:54:20.367 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:54:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:54:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:54:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:54:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:54:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:54:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:54:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:54:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:54:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:54:20.373 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:20.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:20.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:54:20.375 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:54:20.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:54:20.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:54:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:54:20.831 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:54:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:54:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 2, 2020-10-15T16:54:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:54:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [ 2020-10-15T16:54:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:54:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:54:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "" 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.834 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:54:20.835 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:54:20.836 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:54:20.837 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:54:20.838 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:54:20.839 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:54:20.840 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.841 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:54:20.842 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:54:20.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:54:20.844 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:54:20.845 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:54:20.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.846 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:54:20.855 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:54:20.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.855 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:54:20.855 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:54:20.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.856 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:54:20.856 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:54:20.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:54:20.857 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:54:20.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.858 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:54:20.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.858 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:54:20.858 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:54:20.859 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:54:20.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:54:20.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:54:20.861 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:54:20.861 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:54:20.861 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:54:20.861 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:54:20.862 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-10-15T16:55:20.864 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-49 2020-10-15T16:55:20.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:20.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:20.964 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:21.004 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:21.005 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:21.215 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T16:55:21.248 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:55:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:55:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:55:20.907394Z" 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:21.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:55:21.251 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:21.251 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:55:21.252 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:55:21.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:21.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:21.253 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:21.291 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:21.291 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:21.564 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:21.564 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:21.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:21.566 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:21.567 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:21.568 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:21.569 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:21.571 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:21.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:21.573 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:21.573 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:21.573 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:21.573 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:21.573 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:21.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:26.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:26.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:26.577 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:26.617 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:26.618 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:26.894 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:26.903 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:26.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:31.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:31.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:31.906 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:31.947 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:31.948 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:32.207 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:32.216 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:32.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:37.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:37.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:37.220 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:37.264 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:37.265 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:37.542 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:37.551 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:37.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:42.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:42.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:42.555 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:42.873 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:42.874 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:42.874 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:42.874 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:42.874 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:42.874 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:42.874 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:42.875 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:42.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:42.877 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:42.878 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:42.879 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:42.880 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:42.881 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:42.882 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:42.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:47.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:47.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:47.885 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:47.927 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:47.928 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:48.194 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:48.194 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:48.194 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:48.194 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:48.194 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:48.195 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:48.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:48.195 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:48.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:48.195 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:48.195 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:48.196 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:48.197 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:48.197 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:48.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:48.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:48.197 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:48.198 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:48.198 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:48.198 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:48.198 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:48.198 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:48.198 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:48.200 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:48.200 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:48.200 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:48.200 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:48.200 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:48.200 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:48.201 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:48.201 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:48.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:48.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:48.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:48.201 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:48.202 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:48.203 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:48.203 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:48.203 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:48.203 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:48.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:48.203 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:48.204 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:48.204 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:48.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:48.204 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:48.205 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:51.054 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T16:55:51.049+0000 7ffb75ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T16:55:53.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:53.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:53.207 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:53.250 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:53.251 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:53.526 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:55:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:55:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:55:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:55:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:55:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:55:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:55:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:55:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:55:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:55:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:55:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:55:53.535 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:55:53.536 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:55:53.537 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:55:53.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:53.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:53.538 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:53.577 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:53.577 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:53.834 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:55:53.834 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:55:53.835 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:55:53.835 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:55:53.835 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:55:53.835 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:53.835 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:55:53.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:53.836 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:53.837 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:55:53.837 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:53.837 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:53.837 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:55:53.838 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:53.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:53.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:53.838 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:53.839 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:53.840 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:55:53.841 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:53.842 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:53.843 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:55:53.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:53.844 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:55:53.844 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:55:53.844 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:55:53.844 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:53.844 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:55:53.845 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:55:58.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:58.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:55:58.848 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:55:58.888 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:55:58.889 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:55:59.141 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:55:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:55:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:55:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:55:59.144 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:55:59.144 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:59.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:59.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:59.144 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:55:59.144 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:59.145 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:59.145 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:55:59.145 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:59.145 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:55:59.145 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:59.146 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:55:59.146 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:55:59.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:59.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:59.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:59.146 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:59.147 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:55:59.148 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:59.149 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:55:59.150 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:55:59.151 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:56:04.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:04.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:04.154 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:56:04.193 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:04.193 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:04.447 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:56:04.447 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:56:04.447 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:56:04.448 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:56:04.448 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:56:04.448 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:04.448 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:04.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:04.448 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:56:04.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:04.449 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:56:04.449 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:04.449 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:04.449 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:56:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:56:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:56:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:56:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:56:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:04.456 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:56:04.457 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:56:09.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:09.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:09.460 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:56:09.502 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:09.502 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:09.751 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:56:09.751 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:09.752 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:56:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:56:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:56:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:56:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:56:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:56:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:09.760 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:56:09.761 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:56:14.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:14.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:14.763 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:56:14.804 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:14.804 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:15.062 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:56:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:56:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:56:15.063 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:56:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:56:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:56:15.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:56:15.065 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:56:15.066 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:56:15.067 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:15.068 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:56:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:56:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:56:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:56:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:56:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:15.072 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:56:15.076 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:56:20.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:20.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:20.078 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:56:20.121 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:20.121 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:20.363 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:56:20.363 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:56:20.363 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:56:20.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:56:20.365 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:20.366 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:56:20.366 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:20.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:20.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:20.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:20.366 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:56:20.367 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:20.367 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:20.367 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780852.307429_271.1", 2020-10-15T16:56:20.367 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:20.367 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:20.367 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:54:12.307429Z", 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:56:20.368 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:20.369 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:20.370 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:20.371 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:20.371 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:20.371 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:20.371 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:20.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:20.371 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:20.372 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:56:20.373 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602780920.907394_305.1 target=1_1602780852.307429_271.1 2020-10-15T16:56:21.048 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T16:56:21.045+0000 7f56887f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T16:56:25.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:25.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:25.376 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:56:25.418 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:25.418 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:25.678 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:25.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:56:25.680 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:25.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:25.682 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:56:25.683 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:25.684 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:56:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:25.686 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:56:25.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:56:25.687 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:56:25.687 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:56:25.687 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:56:25.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:56:25.687 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:56:25.688 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:56:25.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:56:25.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:25.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:56:25.691 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:56:25.733 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:25.733 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:26.107 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:56:26.108 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:56:26.108 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:56:26.108 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:56:26.108 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:56:26.108 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:26.108 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:56:26.109 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:56:26.111 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.112 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:56:26.113 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:56:26.114 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.116 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:56:26.117 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:56:26.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=803109df-3f66-4c18-857a-2beb852ab289', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:56:26.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=803109df-3f66-4c18-857a-2beb852ab289', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:56:26.135 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=803109df-3f66-4c18-857a-2beb852ab289 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:56:26.201 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:26.202 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:26.226 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:56:26.224+0000 7f8b9a34b0c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:56:26.227 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:56:26.224+0000 7f8b9a34b0c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:56:26.227 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:56:26.224+0000 7f8b9a34b0c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:56:26.227 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:56:26.224+0000 7f8b9a34b0c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:56:26.240 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:56:26.240 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:56:26.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:56:26.240 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:56:26.241 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:56:26.242 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:56:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:56:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:56:26.246 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:56:26.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:56:26.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:56:26.247 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:56:26.328 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:56:26.328 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:56:26.606 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:56:26.698 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:56:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 3, 2020-10-15T16:56:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:56:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [ 2020-10-15T16:56:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:56:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "" 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:56:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:56:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:56:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:56:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "803109df-3f66-4c18-857a-2beb852ab289" 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:56:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:56:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:56:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:56:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:56:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:56:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:56:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:56:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:56:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:56:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:56:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:56:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:56:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:56:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:56:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:56:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:56:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:56:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:56:26.728 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:56:26.729 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "803109df-3f66-4c18-857a-2beb852ab289" 2020-10-15T16:57:27.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:57:27.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:57:27.080 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:57:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:57:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:57:27.310 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:57:27.151+0000 7f827653a0c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:57:27.310 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:57:27.151+0000 7f827653a0c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:57:27.311 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:57:27.151+0000 7f827653a0c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:57:27.311 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:57:27.151+0000 7f827653a0c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:57:27.311 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:57:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:57:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:57:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:57:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:57:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:57:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:57:27.317 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:57:27.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:57:27.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:57:27.318 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:57:27.391 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:57:27.391 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:57:27.673 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:57:27.752 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:57:27.752 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:57:27.752 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 4, 2020-10-15T16:57:27.752 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [ 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: "" 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.753 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:57:27.754 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:57:27.755 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.756 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:57:27.757 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "803109df-3f66-4c18-857a-2beb852ab289" 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:57:27.758 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:57:27.759 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:57:27.760 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:57:27.761 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:57:27.762 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:57:27.763 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.764 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.765 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.766 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:57:27.766 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:57:27.766 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:57:27.766 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:57:27.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.772 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:57:27.772 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:57:27.773 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:57:27.774 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:57:27.775 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:57:27.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:57:27.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:57:27.778 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:57:27.778 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:57:27.778 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:57:27.778 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:57:27.779 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-10-15T16:58:27.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:58:27.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:58:27.783 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:58:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:58:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:58:27.853 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:58:27.849+0000 7fa5c52600c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:58:27.853 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:58:27.849+0000 7fa5c52600c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:58:27.853 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:58:27.849+0000 7fa5c52600c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:58:27.854 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:58:27.849+0000 7fa5c52600c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:58:27.866 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:58:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:58:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:58:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-15T16:58:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-15T16:58:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-15T16:58:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-15T16:58:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-15T16:58:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-15T16:58:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-15T16:58:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-15T16:58:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:58:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:58:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:58:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:58:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:58:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:58:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-15T16:58:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:58:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "compression_type": "random" 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:58:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:58:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-15T16:58:27.873 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:58:27.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:58:27.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:58:27.875 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:58:27.955 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:58:27.955 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:58:28.222 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:58:28.298 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:58:28.299 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:58:28.299 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 5, 2020-10-15T16:58:28.299 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:58:28.299 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [ 2020-10-15T16:58:28.299 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: "" 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:58:28.300 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:58:28.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.301 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:58:28.301 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:58:28.301 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:58:28.301 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:58:28.301 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:58:28.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.303 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:58:28.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:58:28.303 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:58:28.303 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:58:28.303 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.303 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:58:28.304 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:58:28.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:58:28.304 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:58:28.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:58:28.304 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:58:28.305 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:58:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:58:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:58:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:58:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:58:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:58:28.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:58:28.308 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:58:28.308 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:58:28.308 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.308 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:58:28.308 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:58:28.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.310 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:58:28.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:58:28.310 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:58:28.310 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:58:28.310 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.310 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:58:28.311 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:58:28.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:58:28.313 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:58:28.314 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:58:28.314 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:58:28.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:28.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:28.314 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:58:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:58:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:58:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:58:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:58:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:58:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:58:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:58:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:58:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:58:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:58:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:58:28.345 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:58:28.346 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-10-15T16:59:28.348 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-10-15T16:59:28.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:28.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:28.351 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:28.392 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:28.393 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:28.616 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T16:59:28.650 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:59:28.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:28.650 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:28.650 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:59:28.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:55:20.907394Z" 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:28.651 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:59:28.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:28.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:28.652 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:28.652 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:59:28.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:28.652 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:59:28.653 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:59:28.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:28.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:28.655 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:28.696 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:59:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:59:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:59:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:59:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:59:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:59:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:59:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:59:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:59:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:28.985 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:28.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T16:59:28.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T16:59:28.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:28.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:28.988 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:29.027 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:29.027 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:29.277 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:29.277 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T16:59:29.277 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T16:59:29.277 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T16:59:29.278 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T16:59:29.278 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:29.278 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:59:29.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:29.278 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:59:29.279 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:29.280 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:59:29.281 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:59:29.282 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:59:29.282 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:29.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:29.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:29.282 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:59:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:59:29.284 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:29.285 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:29.286 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:29.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T16:59:29.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T16:59:29.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:29.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:29.289 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:29.330 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:29.330 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:29.704 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:29.704 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:59:29.705 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:59:29.705 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:59:29.705 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:59:29.705 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:29.705 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:29.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.706 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:59:29.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:29.706 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:59:29.706 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:29.706 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:29.706 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:29.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:59:29.709 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:29.710 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:29.711 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.712 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:59:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:59:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:59:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.713 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:29.714 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T16:59:29.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:29.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KJNC8K1PAFDOAHQ8', '--secret', 'aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:29.716 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key KJNC8K1PAFDOAHQ8 --secret aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:29.799 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:29.800 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:29.864 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:29.860+0000 7f5d5ab410c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:59:29.865 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:29.860+0000 7f5d5ab410c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:59:29.865 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:29.864+0000 7f5d5ab410c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:59:29.865 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:29.864+0000 7f5d5ab410c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:59:29.873 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:29.874 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7240cad0-7f41-4e13-ae58-d4933a44eb5b", 2020-10-15T16:59:29.874 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "remove", 2020-10-15T16:59:29.874 INFO:teuthology.orchestra.run.smithi038.stdout: "domain_root": "remove.rgw.meta:root", 2020-10-15T16:59:29.874 INFO:teuthology.orchestra.run.smithi038.stdout: "control_pool": "remove.rgw.control", 2020-10-15T16:59:29.874 INFO:teuthology.orchestra.run.smithi038.stdout: "gc_pool": "remove.rgw.log:gc", 2020-10-15T16:59:29.874 INFO:teuthology.orchestra.run.smithi038.stdout: "lc_pool": "remove.rgw.log:lc", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "log_pool": "remove.rgw.log", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-10-15T16:59:29.875 INFO:teuthology.orchestra.run.smithi038.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: "otp_pool": "remove.rgw.otp", 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: "system_key": { 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: "access_key": "KJNC8K1PAFDOAHQ8", 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: "secret_key": "aiKf1fHK0fBwRXsOzDrecHVyfcmy0hXY" 2020-10-15T16:59:29.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_pools": [ 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "default-placement", 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "index_pool": "remove.rgw.buckets.index", 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": { 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD": { 2020-10-15T16:59:29.877 INFO:teuthology.orchestra.run.smithi038.stdout: "data_pool": "remove.rgw.buckets.data" 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: "index_type": 0 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:29.878 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:29.879 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:29.879 INFO:teuthology.orchestra.run.smithi038.stdout: "notif_pool": "remove.rgw.log:notif" 2020-10-15T16:59:29.879 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:29.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:29.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:29.881 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:29.957 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:29.958 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:37.625 INFO:teuthology.orchestra.run.smithi038.stderr:Sending period to new master zone 803109df-3f66-4c18-857a-2beb852ab289 2020-10-15T16:59:37.699 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 6, 2020-10-15T16:59:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:59:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": [ 2020-10-15T16:59:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:59:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:59:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2020-10-15T16:59:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "" 2020-10-15T16:59:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "period_map": { 2020-10-15T16:59:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "zonegroups": [ 2020-10-15T16:59:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "test-zonegroup", 2020-10-15T16:59:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zonegroup", 2020-10-15T16:59:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "api_name": "", 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "is_master": "true", 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames": [], 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "hostnames_s3website": [], 2020-10-15T16:59:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "zones": [ 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone2", 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:59:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7240cad0-7f41-4e13-ae58-d4933a44eb5b", 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "remove", 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [], 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:59:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:59:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:59:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:59:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:59:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone3", 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:59:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone1", 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:59:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "", 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [], 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test-zone4", 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "endpoints": [ 2020-10-15T16:59:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "log_meta": "false", 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "log_data": "true", 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "read_only": "false", 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_type": "pubsub", 2020-10-15T16:59:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from_all": "false", 2020-10-15T16:59:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_from": [ 2020-10-15T16:59:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "test-zone1" 2020-10-15T16:59:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_zone": "" 2020-10-15T16:59:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "placement_targets": [ 2020-10-15T16:59:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default-placement", 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_classes": [ 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "STANDARD" 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "default_placement": "default-placement", 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_policy": { 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "groups": [] 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "short_zone_ids": [ 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 757601199 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7240cad0-7f41-4e13-ae58-d4933a44eb5b", 2020-10-15T16:59:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2347553537 2020-10-15T16:59:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 1274429389 2020-10-15T16:59:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 770630339 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 2077483514 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "period_config": { 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket_quota": { 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:59:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "user_quota": { 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": false, 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "check_on_raw": false, 2020-10-15T16:59:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size": -1, 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max_size_kb": 0, 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max_objects": -1 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_name": "test-realm", 2020-10-15T16:59:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:37.727 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:37.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:37.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:37.729 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:37.773 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:37.774 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:37.793 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:37.789+0000 7f0d3b7a70c0 0 failed reading obj info from .rgw.root:zone_info.669d2e2a-b26c-4898-99f7-fa22d626fe93: (2) No such file or directory 2020-10-15T16:59:37.794 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:37.789+0000 7f0d3b7a70c0 0 WARNING: could not read zone params for zone id=669d2e2a-b26c-4898-99f7-fa22d626fe93 name=test-zone2 2020-10-15T16:59:37.795 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:37.793+0000 7f0d3b7a70c0 0 failed reading obj info from .rgw.root:zone_info.e36fa986-91ea-474b-bd28-fd91c0368de2: (2) No such file or directory 2020-10-15T16:59:37.795 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:37.793+0000 7f0d3b7a70c0 0 WARNING: could not read zone params for zone id=e36fa986-91ea-474b-bd28-fd91c0368de2 name=test-zone4 2020-10-15T16:59:37.801 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:37.801 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:59:37.802 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:59:37.802 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:59:37.802 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:59:37.802 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:37.802 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:37.803 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone2", 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.804 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:37.806 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:59:37.806 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:37.806 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:59:37.806 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.806 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:37.806 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:37.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.808 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:37.809 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone4", 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:59:37.810 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "pubsub", 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "false", 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [ 2020-10-15T16:59:37.811 INFO:teuthology.orchestra.run.smithi183.stdout: "test-zone1" 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:37.812 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:59:37.813 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:37.814 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:59:37.814 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:59:37.814 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:37.814 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:37.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:37.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:37.816 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:37.893 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:37.893 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:37.917 INFO:teuthology.orchestra.run.smithi183.stderr:zone name remove not found in zonegroup test-zonegroup 2020-10-15T16:59:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-10-15T16:59:37.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:37.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:37.924 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:37.962 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:37.962 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:38.003 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:38.000+0000 7f4eec9630c0 0 failed reading obj info from .rgw.root:zone_info.7a2db8d8-0634-4b24-9b71-a341278123d1: (2) No such file or directory 2020-10-15T16:59:38.004 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:38.000+0000 7f4eec9630c0 0 WARNING: could not read zone params for zone id=7a2db8d8-0634-4b24-9b71-a341278123d1 name=test-zone3 2020-10-15T16:59:38.004 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:38.000+0000 7f4eec9630c0 0 failed reading obj info from .rgw.root:zone_info.803109df-3f66-4c18-857a-2beb852ab289: (2) No such file or directory 2020-10-15T16:59:38.005 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:38.000+0000 7f4eec9630c0 0 WARNING: could not read zone params for zone id=803109df-3f66-4c18-857a-2beb852ab289 name=test-zone1 2020-10-15T16:59:38.008 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T16:59:38.004+0000 7f4eec9630c0 0 zone id 7240cad0-7f41-4e13-ae58-d4933a44eb5b is not a part of zonegroup default 2020-10-15T16:59:38.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.021 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:38.059 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:38.060 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:38.122 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:38.121+0000 7fc0d204f0c0 0 WARNING: can't generate connection for zone 7240cad0-7f41-4e13-ae58-d4933a44eb5b id remove: no endpoints defined 2020-10-15T16:59:38.431 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:38.431 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:38.431 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 7, 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": [ 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: "" 2020-10-15T16:59:38.432 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: "period_map": { 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: "zonegroups": [ 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:59:38.433 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:59:38.434 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone2", 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:59:38.435 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:38.436 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:38.437 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:38.438 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:59:38.439 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:38.440 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:38.441 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone4", 2020-10-15T16:59:38.442 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:38.442 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:59:38.443 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.444 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:38.444 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:38.444 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:38.444 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:38.444 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "pubsub", 2020-10-15T16:59:38.444 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "false", 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [ 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: "test-zone1" 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.445 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:38.446 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.447 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.475 INFO:teuthology.orchestra.run.smithi183.stdout: "short_zone_ids": [ 2020-10-15T16:59:38.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.475 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:38.475 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 757601199 2020-10-15T16:59:38.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 1274429389 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:38.476 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 770630339 2020-10-15T16:59:38.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.477 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:38.477 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 2077483514 2020-10-15T16:59:38.477 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.477 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:38.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.478 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:59:38.478 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:38.478 INFO:teuthology.orchestra.run.smithi183.stdout: "period_config": { 2020-10-15T16:59:38.478 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:59:38.478 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:59:38.479 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:59:38.479 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:59:38.479 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:59:38.479 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:59:38.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.479 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:59:38.480 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:59:38.480 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:59:38.480 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:59:38.480 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:59:38.480 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:59:38.480 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.481 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:38.481 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_name": "test-realm", 2020-10-15T16:59:38.481 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:59:38.481 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:38.482 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-10-15T16:59:38.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.484 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:38.563 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:38.563 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:38.593 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:38.593 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "62a17e79-f251-4c7a-a7d6-3bd6598d8342", 2020-10-15T16:59:38.593 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "remove_zg", 2020-10-15T16:59:38.593 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "remove_zg", 2020-10-15T16:59:38.593 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "false", 2020-10-15T16:59:38.594 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [], 2020-10-15T16:59:38.594 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:59:38.594 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:59:38.594 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "", 2020-10-15T16:59:38.594 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [], 2020-10-15T16:59:38.594 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [], 2020-10-15T16:59:38.595 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "", 2020-10-15T16:59:38.595 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:38.595 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:59:38.595 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:59:38.595 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.595 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:38.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.597 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:38.675 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:38.675 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:38.734 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "536e3bfe-f893-4928-b241-a677d54e26d6", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "remove", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "domain_root": "remove.rgw.meta:root", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "control_pool": "remove.rgw.control", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "gc_pool": "remove.rgw.log:gc", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "lc_pool": "remove.rgw.log:lc", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "log_pool": "remove.rgw.log", 2020-10-15T16:59:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-10-15T16:59:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-10-15T16:59:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-10-15T16:59:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-10-15T16:59:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-10-15T16:59:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-10-15T16:59:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "otp_pool": "remove.rgw.otp", 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "system_key": { 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "access_key": "", 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "secret_key": "" 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_pools": [ 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "default-placement", 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "index_pool": "remove.rgw.buckets.index", 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": { 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD": { 2020-10-15T16:59:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "data_pool": "remove.rgw.buckets.data" 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "index_type": 0 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "notif_pool": "remove.rgw.log:notif" 2020-10-15T16:59:38.740 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:38.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:38.742 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:38.825 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:38.826 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:39.127 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:39.127 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:39.127 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 8, 2020-10-15T16:59:39.127 INFO:teuthology.orchestra.run.smithi183.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": [ 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: "" 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.128 INFO:teuthology.orchestra.run.smithi183.stdout: "period_map": { 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: "zonegroups": [ 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "62a17e79-f251-4c7a-a7d6-3bd6598d8342", 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "remove_zg", 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "remove_zg", 2020-10-15T16:59:39.129 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "false", 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [], 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "536e3bfe-f893-4928-b241-a677d54e26d6", 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.130 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "536e3bfe-f893-4928-b241-a677d54e26d6", 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "remove", 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [], 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "false", 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.131 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:59:39.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.133 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.134 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.135 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone2", 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.136 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.137 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:59:39.138 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.139 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:59:39.140 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.141 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:39.141 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.141 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.141 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.141 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.141 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.166 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.166 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.166 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.166 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone4", 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.167 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "pubsub", 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "false", 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [ 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: "test-zone1" 2020-10-15T16:59:39.168 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:59:39.169 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:39.170 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: "short_zone_ids": [ 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.171 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "536e3bfe-f893-4928-b241-a677d54e26d6", 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 3349334129 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 757601199 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 1274429389 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 770630339 2020-10-15T16:59:39.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 2077483514 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:59:39.174 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "period_config": { 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:59:39.175 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:59:39.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.177 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:39.177 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_name": "test-realm", 2020-10-15T16:59:39.177 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:59:39.177 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:39.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.179 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:39.263 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:39.264 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:39.315 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:39.313+0000 7f268a2290c0 0 zone id 536e3bfe-f893-4928-b241-a677d54e26d6 is not a part of zonegroup test-zonegroup 2020-10-15T16:59:39.316 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:39.313+0000 7f268a2290c0 0 zone id 536e3bfe-f893-4928-b241-a677d54e26d6 is not a part of zonegroup default 2020-10-15T16:59:39.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.338 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:39.418 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:39.419 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:39.461 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T16:59:39.457+0000 7f89c8dca0c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-10-15T16:59:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-10-15T16:59:39.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.468 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:39.546 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:39.546 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:39.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T16:59:39.583 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T16:59:39.665 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:39.665 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:39.951 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T16:59:39.951 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:39.951 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 9, 2020-10-15T16:59:39.951 INFO:teuthology.orchestra.run.smithi183.stdout: "predecessor_uuid": "b84643c0-8302-4cfb-a70b-7af9afa8e893", 2020-10-15T16:59:39.952 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": [ 2020-10-15T16:59:39.952 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:59:39.952 INFO:teuthology.orchestra.run.smithi183.stdout: "1_1602780771.905271_175.1", 2020-10-15T16:59:39.952 INFO:teuthology.orchestra.run.smithi183.stdout: "", 2020-10-15T16:59:39.952 INFO:teuthology.orchestra.run.smithi183.stdout: "" 2020-10-15T16:59:39.952 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: "period_map": { 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: "zonegroups": [ 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "test-zonegroup", 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zonegroup", 2020-10-15T16:59:39.953 INFO:teuthology.orchestra.run.smithi183.stdout: "api_name": "", 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: "is_master": "true", 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames": [], 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: "hostnames_s3website": [], 2020-10-15T16:59:39.954 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: "zones": [ 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone2", 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8000" 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.955 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.956 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone3", 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8001" 2020-10-15T16:59:39.957 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.958 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone1", 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.959 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi183.front.sepia.ceph.com:8000" 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "", 2020-10-15T16:59:39.960 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "true", 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [], 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: "id": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "test-zone4", 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: "endpoints": [ 2020-10-15T16:59:39.961 INFO:teuthology.orchestra.run.smithi183.stdout: "http://smithi038.front.sepia.ceph.com:8001" 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: "log_meta": "false", 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: "log_data": "true", 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_index_max_shards": 11, 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: "read_only": "false", 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_type": "pubsub", 2020-10-15T16:59:39.962 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from_all": "false", 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_from": [ 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: "test-zone1" 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: "redirect_zone": "" 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.963 INFO:teuthology.orchestra.run.smithi183.stdout: "placement_targets": [ 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "default-placement", 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: "tags": [], 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: "storage_classes": [ 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: "STANDARD" 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:39.964 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: "default_placement": "default-placement", 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_policy": { 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: "groups": [] 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.965 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2020-10-15T16:59:39.992 INFO:teuthology.orchestra.run.smithi183.stdout: "short_zone_ids": [ 2020-10-15T16:59:39.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.992 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 757601199 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 1274429389 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 770630339 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "e36fa986-91ea-474b-bd28-fd91c0368de2", 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: "val": 2077483514 2020-10-15T16:59:39.994 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zonegroup": "test-zonegroup", 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: "master_zone": "803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: "period_config": { 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket_quota": { 2020-10-15T16:59:39.995 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: "user_quota": { 2020-10-15T16:59:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: "enabled": false, 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: "check_on_raw": false, 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size": -1, 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: "max_size_kb": 0, 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: "max_objects": -1 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_id": "9e20c2cc-422b-4644-9bb7-b97bb5e18239", 2020-10-15T16:59:39.998 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_name": "test-realm", 2020-10-15T16:59:39.998 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T16:59:39.998 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T16:59:39.999 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-10-15T16:59:40.000 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-50 2020-10-15T16:59:40.513 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-51 2020-10-15T16:59:40.902 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-52 2020-10-15T16:59:40.983 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-53 2020-10-15T16:59:41.017 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-54 2020-10-15T16:59:41.088 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-55 2020-10-15T16:59:43.350 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-10-15T16:59:43.351 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-56 2020-10-15T16:59:43.399 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-57 2020-10-15T16:59:43.471 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-58 2020-10-15T16:59:43.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:43.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:43.524 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:43.558 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:43.559 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:43.783 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T16:59:43.816 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T16:59:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:59:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602781183.505226_417.1", 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T16:59:43.505226Z" 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T16:59:43.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T16:59:43.819 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T16:59:43.819 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T16:59:43.819 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T16:59:43.819 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T16:59:43.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T16:59:43.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:43.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:43.821 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:43.864 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:43.865 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:44.143 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:44.143 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:59:44.143 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:59:44.144 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:59:44.144 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:59:44.144 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:44.144 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:44.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:44.144 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:59:44.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:44.145 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:59:44.145 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:44.145 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:44.145 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:59:44.145 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:44.146 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:59:44.146 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:44.146 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:59:44.146 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:44.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:44.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:44.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:44.147 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:59:44.147 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:44.147 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:44.147 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:44.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:59:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:59:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:59:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:59:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:59:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:44.153 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:44.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T16:59:49.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:49.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:49.156 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:49.198 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:49.198 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:49.477 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:59:49.477 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:59:49.477 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:59:49.477 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:59:49.477 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:49.478 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:49.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:49.478 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:59:49.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:49.478 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:59:49.478 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:59:49.479 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:49.480 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:59:49.485 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:59:49.486 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:49.486 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:49.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T16:59:54.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:54.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:54.489 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:54.531 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:54.531 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T16:59:54.824 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T16:59:54.824 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T16:59:54.824 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T16:59:54.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:54.826 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T16:59:54.826 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:54.826 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:54.826 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T16:59:54.827 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:54.827 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T16:59:54.827 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:54.827 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T16:59:54.827 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:54.828 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:54.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:54.830 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:54.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:54.831 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:54.831 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:54.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:54.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:54.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T16:59:54.832 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T16:59:54.833 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T16:59:54.834 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T16:59:54.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T16:59:54.834 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T16:59:54.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T16:59:59.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:59.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T16:59:59.837 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T16:59:59.878 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T16:59:59.878 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:00.157 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:00:00.157 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:00:00.158 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:00:00.158 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:00:00.158 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:00:00.158 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:00.158 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:00.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:00:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:00:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:00:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:00:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:00.166 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:00:00.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:05.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:05.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:05.169 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:05.214 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:05.214 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:05.486 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:00:05.486 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:00:05.486 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:00:05.486 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:00:05.486 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:00:05.487 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:05.488 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:05.489 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:05.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:05.491 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:00:05.492 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:05.493 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:00:05.494 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:00:05.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:05.495 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:00:05.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:10.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:10.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:10.499 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:10.538 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:10.539 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:10.822 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:00:10.822 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:10.823 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:10.824 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:10.825 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:10.826 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:10.827 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:10.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:10.829 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:00:10.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:10.831 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:00:10.831 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:00:10.831 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:00:10.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:10.831 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:00:10.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:15.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:15.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:15.834 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:15.874 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:15.875 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:16.152 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:00:16.153 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:00:16.153 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:00:16.153 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:00:16.153 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:00:16.153 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:16.154 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:16.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781183.505226_417.1", 2020-10-15T17:00:16.156 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:59:43.505226Z", 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:16.158 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:00:16.159 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:00:16.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:00:16.161 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:00:16.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T17:00:16.163 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T17:00:16.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:16.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:16.164 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:16.203 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:16.203 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:16.457 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:16.457 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:16.457 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:16.457 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:16.458 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:16.458 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:16.458 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:16.459 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:16.461 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:16.462 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:16.463 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:16.464 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:16.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:16.466 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:16.466 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:16.466 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:16.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:16.466 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:16.467 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:21.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:21.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:21.470 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:21.510 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:21.511 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:21.767 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:21.781 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:21.782 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:26.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:26.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:26.785 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:26.824 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:26.824 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:27.278 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:27.279 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:27.279 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:27.279 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:27.279 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:27.279 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:27.279 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:27.280 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:27.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:27.282 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:27.283 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:27.284 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:27.285 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:27.286 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:27.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:27.287 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:27.287 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:27.287 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:27.287 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:27.287 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:27.288 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:32.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:32.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:32.291 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:32.330 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:32.330 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:32.598 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:32.599 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:32.599 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:32.599 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:32.599 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:32.599 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:32.600 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:32.601 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:32.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:32.604 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:32.604 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:32.604 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:32.604 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:32.604 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:32.604 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:32.605 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:32.606 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:32.607 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:32.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:32.608 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:32.609 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:37.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:37.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:37.611 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:37.651 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:37.651 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:37.911 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:37.912 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:37.912 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:37.912 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:37.912 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:37.912 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:37.912 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:37.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:37.915 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:37.916 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:37.917 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:37.918 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:37.919 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:37.920 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:37.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:37.920 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:37.920 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:37.920 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:37.920 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:37.921 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:37.922 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:42.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:42.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:42.925 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:42.964 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:42.965 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:43.226 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:43.227 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:43.227 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:43.228 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:43.228 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:43.228 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:43.228 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:43.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:43.229 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:43.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:43.229 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:43.230 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:43.230 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:43.230 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:43.231 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:43.231 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:43.231 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:43.231 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:43.232 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:43.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:43.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:43.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:43.233 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:43.233 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:43.234 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:43.234 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:43.234 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:43.234 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:43.235 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:43.235 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:43.235 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:43.235 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:43.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:43.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:43.236 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:43.236 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:43.237 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:43.237 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:43.237 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:43.237 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:43.237 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:43.238 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:43.238 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:43.238 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:43.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:43.239 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:43.240 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:43.241 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:43.241 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:43.241 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:43.241 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:43.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:43.242 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:43.242 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:43.242 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:43.243 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:43.243 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:43.245 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:48.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:48.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:48.247 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:48.287 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:48.287 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:48.544 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:48.544 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:48.545 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:48.545 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:48.545 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:48.545 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:48.545 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:48.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:48.546 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:48.547 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:48.547 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:48.547 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:48.547 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:48.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:48.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:48.548 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:48.550 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:48.551 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:48.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:48.553 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:48.554 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:53.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:53.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:53.557 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:53.596 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:53.596 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:53.848 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:53.849 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:53.849 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:53.849 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:53.849 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:53.849 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:53.850 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:53.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:53.850 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:53.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:53.850 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:53.850 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:53.851 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:53.852 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:53.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:53.854 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:53.855 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:53.856 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:53.857 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:53.858 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:53.858 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:00:58.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:58.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:00:58.860 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:00:58.899 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:00:58.900 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:00:59.134 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:00:59.134 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:00:59.134 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:00:59.134 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:00:59.135 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:00:59.135 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:00:59.135 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:59.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:59.135 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:00:59.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:00:59.136 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:59.137 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:00:59.138 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:59.139 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:00:59.140 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:00:59.141 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:00:59.142 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:00:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:00:59.143 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:00:59.143 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:00:59.144 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:04.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:04.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:04.146 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:04.188 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:04.188 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:04.449 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:04.450 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:04.451 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:04.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:04.453 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:04.454 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:04.455 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:04.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:04.457 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:04.457 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:04.457 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:04.457 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:04.457 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:04.458 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:04.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:04.459 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:04.459 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:04.459 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:04.459 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:04.459 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:04.460 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:09.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:09.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:09.463 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:09.506 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:09.507 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:09.755 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:09.763 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:09.763 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:09.763 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:09.763 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:09.763 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:09.763 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:09.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:09.765 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:09.765 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:14.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:14.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:14.767 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:14.809 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:14.809 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:15.068 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:15.069 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:15.070 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:15.071 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:15.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:15.073 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:15.074 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:15.075 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:15.076 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:15.077 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:15.078 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:15.079 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:15.080 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:15.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:15.081 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:15.082 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:20.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:20.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:20.085 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:20.124 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:20.124 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:20.379 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:20.379 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:20.380 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:20.381 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:20.382 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:20.383 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:20.384 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:20.386 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:20.387 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:20.388 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:20.388 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:20.388 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:20.388 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:20.388 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:20.389 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:25.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:25.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:25.392 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:25.432 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:25.433 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:25.701 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:25.701 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:25.702 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:25.702 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:25.702 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:25.702 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:25.702 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:25.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:25.703 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:25.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:25.703 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:25.703 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:25.704 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:25.705 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:25.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:25.707 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:25.708 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:25.709 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:25.710 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:25.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:25.711 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:25.712 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:30.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:30.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:30.715 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:30.754 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:30.754 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:31.019 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:31.020 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:31.020 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:31.020 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:31.020 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:31.020 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:31.020 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:31.021 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:31.025 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:31.025 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:31.025 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:31.025 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:31.025 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:31.025 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:31.026 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:31.027 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:31.028 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:31.029 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:31.029 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:31.030 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:36.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:36.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:36.032 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:36.074 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:36.075 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:36.319 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:36.320 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:36.320 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:36.320 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:36.320 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:36.320 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:36.320 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:36.321 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:36.322 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:36.325 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:36.326 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:36.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:36.328 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:36.329 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:36.329 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:41.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:41.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:41.332 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:41.371 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:41.371 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:41.619 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:41.620 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:41.620 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:41.620 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:41.620 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:41.620 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:41.620 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:41.621 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:41.622 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:41.622 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:41.622 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:41.622 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:41.622 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:41.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780920.907394_305.1", 2020-10-15T17:01:41.623 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:55:20.907394Z", 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:41.624 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:41.625 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:41.626 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:41.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:41.628 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:41.629 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1602781183.505226_417.1 target=1_1602780920.907394_305.1 2020-10-15T17:01:46.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:46.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:46.632 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:46.673 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:46.673 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:46.945 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:01:46.946 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:01:46.946 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:01:46.946 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:01:46.946 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:01:46.947 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:46.947 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:46.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:46.947 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:01:46.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:46.947 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:46.948 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:46.949 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:46.949 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:46.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:46.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:46.949 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:01:46.949 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602781183.505226_417.1", 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:59:43.505226Z", 2020-10-15T17:01:46.950 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:01:46.951 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:46.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:46.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:46.951 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:01:46.951 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:46.951 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:46.952 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:01:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:01:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:01:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:01:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:01:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:46.955 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:01:46.956 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T17:01:46.957 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T17:01:46.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:46.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:46.958 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:46.999 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:47.370 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:01:47.371 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:01:47.371 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:01:47.371 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:01:47.371 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:01:47.371 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:47.372 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:01:47.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781183.505226_417.1", 2020-10-15T17:01:47.374 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:59:43.505226Z", 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:47.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:47.376 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:47.377 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:01:47.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:47.379 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:01:47.380 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-15T17:01:47.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'iybjjr-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T17:01:47.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'iybjjr-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T17:01:47.382 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket iybjjr-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T17:01:47.423 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:47.424 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:47.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'iybjjr-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T17:01:47.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'iybjjr-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T17:01:47.770 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket iybjjr-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T17:01:47.848 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:47.849 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:48.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'iybjjr-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T17:01:48.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'iybjjr-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-15T17:01:48.192 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket iybjjr-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-15T17:01:48.274 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:48.274 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:48.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:48.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:48.619 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:48.703 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:48.704 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:49.005 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.1", 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000078", 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.history", 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.8-iybjjr-8", 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000070", 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000104", 2020-10-15T17:01:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000026", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.2-iybjjr-45", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.2", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000028", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.5-iybjjr-54", 2020-10-15T17:01:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.5-iybjjr-48", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000040", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.0", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000015", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000069", 2020-10-15T17:01:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.1-iybjjr-50", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000095", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000003", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000047", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000035", 2020-10-15T17:01:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000037", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.26-iybjjr-26", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000024", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.1-iybjjr-1", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000000", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000031", 2020-10-15T17:01:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.log.69fb12ff-ae85-466c-bb2d-3018777b1732.1", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.3", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.7-iybjjr-56", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000076", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000113", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000005", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000011", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000058", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000052", 2020-10-15T17:01:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.5-iybjjr-5", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000088", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000080", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000090", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000110", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000096", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000087", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000008", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.2", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.32-iybjjr-32", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.19-iybjjr-19", 2020-10-15T17:01:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000006", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000029", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000089", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000022", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "bilog.trim", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000016", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T17:01:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000041", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.18-iybjjr-18", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.1", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000018", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000092", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.0", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.6-iybjjr-55", 2020-10-15T17:01:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000014", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.33-iybjjr-33", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000112", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000007", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000021", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000064", 2020-10-15T17:01:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.20-iybjjr-20", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000071", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000074", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.3", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.14-iybjjr-14", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000081", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000009", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000121", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.1", 2020-10-15T17:01:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1-iybjjr-42", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000125", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000082", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.1", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000105", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000059", 2020-10-15T17:01:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.9-iybjjr-9", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000077", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000032", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.15-iybjjr-15", 2020-10-15T17:01:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.24-iybjjr-24", 2020-10-15T17:01:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.29-iybjjr-29", 2020-10-15T17:01:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000053", 2020-10-15T17:01:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000091", 2020-10-15T17:01:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.27-iybjjr-27", 2020-10-15T17:01:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.2", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.3-iybjjr-3", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000065", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000083", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000010", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.3.wlyG8duCtRd.0", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.2", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000045", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000002", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.0", 2020-10-15T17:01:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.35-iybjjr-35", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000116", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000034", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.4-iybjjr-47", 2020-10-15T17:01:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000101", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.1", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000079", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000049", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000117", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000044", 2020-10-15T17:01:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.16-iybjjr-16", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.9-iybjjr-58", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000066", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000068", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.40-iybjjr-40", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000085", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.41-iybjjr-41", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.22-iybjjr-22", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000073", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000038", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T17:01:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.3", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000118", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000036", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000103", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.3", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.31-iybjjr-31", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000119", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.1.rYuc5URHBJp.0", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000098", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.11-iybjjr-11", 2020-10-15T17:01:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.0", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000027", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.25-iybjjr-25", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000019", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000039", 2020-10-15T17:01:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.23-iybjjr-23", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000100", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.8-iybjjr-57", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000093", 2020-10-15T17:01:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.3-iybjjr-46", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.0./ZaFN+VGruF.0", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000004", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000063", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000122", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000057", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.3-iybjjr-52", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000054", 2020-10-15T17:01:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000114", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.0", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.2-iybjjr-51", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.34-iybjjr-34", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.0", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000012", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000084", 2020-10-15T17:01:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000043", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.3", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.1", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.1-iybjjr-44", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000111", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000048", 2020-10-15T17:01:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000020", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.10-iybjjr-10", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000099", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000056", 2020-10-15T17:01:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000072", 2020-10-15T17:01:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000062", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000115", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000033", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2-iybjjr-43", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.13-iybjjr-13", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.17-iybjjr-17", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000025", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000123", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000108", 2020-10-15T17:01:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000094", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.7-iybjjr-7", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000060", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000109", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000013", 2020-10-15T17:01:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.2.Qy+OflPVlb1.0", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.0", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5352.1-iybjjr-49", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000051", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.2-iybjjr-2", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000106", 2020-10-15T17:01:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000017", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.28-iybjjr-28", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000023", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.3", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000107", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000046", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.6-iybjjr-6", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.2", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.2", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000067", 2020-10-15T17:01:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.30-iybjjr-30", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000086", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.37-iybjjr-37", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.39-iybjjr-39", 2020-10-15T17:01:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.1", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.12-iybjjr-12", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.3", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.4-iybjjr-53", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.21-iybjjr-21", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.36-iybjjr-36", 2020-10-15T17:01:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000050", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000102", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000124", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000001", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000126", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000055", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000061", 2020-10-15T17:01:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.4-iybjjr-4", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000097", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000042", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.38-iybjjr-38", 2020-10-15T17:01:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000075", 2020-10-15T17:01:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000120", 2020-10-15T17:01:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000030", 2020-10-15T17:01:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.2" 2020-10-15T17:01:49.048 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:01:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T17:01:49.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:49.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:49.050 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:49.089 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:49.089 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:49.415 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T17:01:49.415 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000078", 2020-10-15T17:01:49.415 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.history", 2020-10-15T17:01:49.415 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.1", 2020-10-15T17:01:49.415 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.8-iybjjr-8", 2020-10-15T17:01:49.416 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000070", 2020-10-15T17:01:49.416 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000104", 2020-10-15T17:01:49.417 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000026", 2020-10-15T17:01:49.417 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.2-iybjjr-45", 2020-10-15T17:01:49.417 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.2", 2020-10-15T17:01:49.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:49.417 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000028", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.1", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.5-iybjjr-54", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.5-iybjjr-48", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000040", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.0", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000015", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000069", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000095", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000003", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000047", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:49.419 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000035", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000037", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.26-iybjjr-26", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000024", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.1-iybjjr-1", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000000", 2020-10-15T17:01:49.420 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000031", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.log.69fb12ff-ae85-466c-bb2d-3018777b1732.1", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.3", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:49.421 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000076", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000113", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000005", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000011", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000058", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000052", 2020-10-15T17:01:49.422 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.5-iybjjr-5", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000088", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000080", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000090", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000110", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000096", 2020-10-15T17:01:49.423 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.0", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000087", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000008", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.19-iybjjr-19", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000006", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000029", 2020-10-15T17:01:49.424 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000089", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000022", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "bilog.trim", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000016", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.2", 2020-10-15T17:01:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000041", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.0.ZPzghyrsYVE.0", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.18-iybjjr-18", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.1", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000018", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000092", 2020-10-15T17:01:49.426 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.0", 2020-10-15T17:01:49.427 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.2", 2020-10-15T17:01:49.427 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:49.427 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:49.427 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000014", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000112", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000007", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000021", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.3.fZE01fEJ2dJ.0", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000064", 2020-10-15T17:01:49.428 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.20-iybjjr-20", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000071", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000074", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.3", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:49.429 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.14-iybjjr-14", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000081", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000009", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000121", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.1", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1-iybjjr-42", 2020-10-15T17:01:49.430 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000125", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000082", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.1", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000105", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000059", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.9-iybjjr-9", 2020-10-15T17:01:49.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000077", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000032", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.15-iybjjr-15", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.24-iybjjr-24", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000053", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000091", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.2", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.3-iybjjr-3", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000065", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000083", 2020-10-15T17:01:49.433 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000010", 2020-10-15T17:01:49.434 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:49.434 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:49.434 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.2", 2020-10-15T17:01:49.434 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:49.434 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000045", 2020-10-15T17:01:49.434 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000002", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000116", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.1.jmasnQfpnqb.0", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000034", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:49.435 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.4-iybjjr-47", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000101", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.1", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000079", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000049", 2020-10-15T17:01:49.436 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000117", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000044", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.16-iybjjr-16", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000066", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000068", 2020-10-15T17:01:49.437 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.40-iybjjr-40", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000085", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.41-iybjjr-41", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.22-iybjjr-22", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000073", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000038", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.3", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000118", 2020-10-15T17:01:49.438 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000036", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000103", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.3", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.31-iybjjr-31", 2020-10-15T17:01:49.439 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000119", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000098", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.11-iybjjr-11", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.0", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000027", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.25-iybjjr-25", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status", 2020-10-15T17:01:49.440 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000019", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000039", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.23-iybjjr-23", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000100", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.8-iybjjr-57", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:49.441 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000093", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.3-iybjjr-46", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000004", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000063", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000122", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000057", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:49.442 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000054", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000114", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.0", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.2-iybjjr-51", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.3", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.34-iybjjr-34", 2020-10-15T17:01:49.443 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.0", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000012", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000084", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000043", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.7a2db8d8-0634-4b24-9b71-a341278123d1.1", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:49.444 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.1-iybjjr-44", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000111", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000048", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000020", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.10-iybjjr-10", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000099", 2020-10-15T17:01:49.445 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000056", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000072", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000062", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000115", 2020-10-15T17:01:49.446 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000033", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2-iybjjr-43", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.13-iybjjr-13", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.17-iybjjr-17", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000025", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:49.447 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000123", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000108", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000094", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.7-iybjjr-7", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000060", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000109", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000013", 2020-10-15T17:01:49.448 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5352.1-iybjjr-49", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000051", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.2-iybjjr-2", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000106", 2020-10-15T17:01:49.449 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.0", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000017", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.28-iybjjr-28", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000023", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.7a2db8d8-0634-4b24-9b71-a341278123d1.3", 2020-10-15T17:01:49.450 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000107", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000046", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.6-iybjjr-6", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.2", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:49.451 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.2", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000067", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000086", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.37-iybjjr-37", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:49.452 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.39-iybjjr-39", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.12-iybjjr-12", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.21-iybjjr-21", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.2.gZC10c1rHJo.0", 2020-10-15T17:01:49.453 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.36-iybjjr-36", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000050", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000102", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000124", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000001", 2020-10-15T17:01:49.454 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000126", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000055", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000061", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.4-iybjjr-4", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000097", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.3", 2020-10-15T17:01:49.455 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000042", 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.7a2db8d8-0634-4b24-9b71-a341278123d1:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000075", 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000120", 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000030" 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T17:01:49.456 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T17:01:49.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:49.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:49.458 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:49.500 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:49.500 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:49.799 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:01:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.1", 2020-10-15T17:01:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000078", 2020-10-15T17:01:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.history", 2020-10-15T17:01:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.1", 2020-10-15T17:01:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.8-iybjjr-8", 2020-10-15T17:01:49.800 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000070", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000104", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000026", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.2-iybjjr-45", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000028", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.1", 2020-10-15T17:01:49.801 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.5-iybjjr-48", 2020-10-15T17:01:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000040", 2020-10-15T17:01:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.0", 2020-10-15T17:01:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000015", 2020-10-15T17:01:49.802 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000069", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000095", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000003", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000047", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000035", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000037", 2020-10-15T17:01:49.803 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.26-iybjjr-26", 2020-10-15T17:01:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000024", 2020-10-15T17:01:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.1-iybjjr-1", 2020-10-15T17:01:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000000", 2020-10-15T17:01:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000031", 2020-10-15T17:01:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:49.804 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.log.69fb12ff-ae85-466c-bb2d-3018777b1732.1", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.3", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:49.805 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000076", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000113", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000005", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000011", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000058", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000052", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.5-iybjjr-5", 2020-10-15T17:01:49.806 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000088", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000080", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000090", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000110", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000096", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.0", 2020-10-15T17:01:49.807 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000087", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000008", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.2", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.32-iybjjr-32", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.19-iybjjr-19", 2020-10-15T17:01:49.808 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000006", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000029", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000089", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000022", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "bilog.trim", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000016", 2020-10-15T17:01:49.809 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.2", 2020-10-15T17:01:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000041", 2020-10-15T17:01:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.18-iybjjr-18", 2020-10-15T17:01:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000018", 2020-10-15T17:01:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000092", 2020-10-15T17:01:49.810 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.6-iybjjr-55", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.2", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000014", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000112", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000007", 2020-10-15T17:01:49.811 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000021", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000064", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.20-iybjjr-20", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000071", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000074", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.14-iybjjr-14", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000081", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000009", 2020-10-15T17:01:49.813 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000121", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.1", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.1-iybjjr-42", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000125", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000082", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.1", 2020-10-15T17:01:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000105", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000059", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.9-iybjjr-9", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000077", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000032", 2020-10-15T17:01:49.815 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.15-iybjjr-15", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.24-iybjjr-24", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.29-iybjjr-29", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000053", 2020-10-15T17:01:49.816 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000091", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.3-iybjjr-3", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000065", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000083", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000010", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.1.NNElUuImxwW.0", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.2", 2020-10-15T17:01:49.817 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000045", 2020-10-15T17:01:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000002", 2020-10-15T17:01:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.0", 2020-10-15T17:01:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.35-iybjjr-35", 2020-10-15T17:01:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000116", 2020-10-15T17:01:49.818 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000034", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.4-iybjjr-47", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000101", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.1", 2020-10-15T17:01:49.819 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000079", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000049", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000117", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000044", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:49.820 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.16-iybjjr-16", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.9-iybjjr-58", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000066", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.2.NQ9TpgsUFKX.0", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000068", 2020-10-15T17:01:49.821 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.40-iybjjr-40", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000085", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.41-iybjjr-41", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.22-iybjjr-22", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000073", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000038", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93", 2020-10-15T17:01:49.822 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.3", 2020-10-15T17:01:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000118", 2020-10-15T17:01:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.3.MjMR7Cx9jAl.0", 2020-10-15T17:01:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:49.823 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000036", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000103", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.3", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000119", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000098", 2020-10-15T17:01:49.824 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.11-iybjjr-11", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.log.0c649eb8-3e99-4d22-800d-adf2e2823b92.0", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status.shard.0", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000027", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.25-iybjjr-25", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "mdlog.sync-status", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000019", 2020-10-15T17:01:49.825 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000039", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.23-iybjjr-23", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000100", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:49.826 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000093", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.3-iybjjr-46", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000004", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000063", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000122", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.log.b84643c0-8302-4cfb-a70b-7af9afa8e893.1", 2020-10-15T17:01:49.827 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000057", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5520.3-iybjjr-52", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000054", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000114", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.0", 2020-10-15T17:01:49.828 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.3", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000012", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000084", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000043", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.3", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5150.1-iybjjr-44", 2020-10-15T17:01:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000111", 2020-10-15T17:01:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000048", 2020-10-15T17:01:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000020", 2020-10-15T17:01:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:49.830 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.10-iybjjr-10", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000099", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000056", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000072", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000062", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000115", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000033", 2020-10-15T17:01:49.831 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-669d2e2a-b26c-4898-99f7-fa22d626fe93.5107.2-iybjjr-43", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.13-iybjjr-13", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.0.aeo5Oi9TPRK.0", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.17-iybjjr-17", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000025", 2020-10-15T17:01:49.832 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000123", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000108", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000094", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.7-iybjjr-7", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000060", 2020-10-15T17:01:49.833 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000109", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000013", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.0", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.5352.1-iybjjr-49", 2020-10-15T17:01:49.834 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000051", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.2-iybjjr-2", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000106", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.0", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000017", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-17-803109df-3f66-4c18-857a-2beb852ab289.5352.1-iybjjr-49", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000023", 2020-10-15T17:01:49.835 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000107", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000046", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.6-iybjjr-6", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.full-sync.index.2", 2020-10-15T17:01:49.836 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:49.837 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:49.837 INFO:teuthology.orchestra.run.smithi183.stdout: "data_log.2", 2020-10-15T17:01:49.837 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000067", 2020-10-15T17:01:49.837 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:49.837 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:49.837 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:49.838 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000086", 2020-10-15T17:01:49.838 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.37-iybjjr-37", 2020-10-15T17:01:49.838 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:49.838 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:49.838 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:49.838 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.39-iybjjr-39", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.1", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.12-iybjjr-12", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.669d2e2a-b26c-4898-99f7-fa22d626fe93.3", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.21-iybjjr-21", 2020-10-15T17:01:49.839 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.36-iybjjr-36", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000050", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000102", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000124", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000001", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000126", 2020-10-15T17:01:49.840 INFO:teuthology.orchestra.run.smithi183.stdout: "meta.log.0c649eb8-3e99-4d22-800d-adf2e2823b92.1", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000055", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000061", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.4-iybjjr-4", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000097", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.3", 2020-10-15T17:01:49.841 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "bucket.sync-status.669d2e2a-b26c-4898-99f7-fa22d626fe93:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000042", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "2020-10-15-16-803109df-3f66-4c18-857a-2beb852ab289.4346.38-iybjjr-38", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000075", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000120", 2020-10-15T17:01:49.842 INFO:teuthology.orchestra.run.smithi183.stdout: "obj_delete_at_hint.0000000030", 2020-10-15T17:01:49.843 INFO:teuthology.orchestra.run.smithi183.stdout: "datalog.sync-status.shard.669d2e2a-b26c-4898-99f7-fa22d626fe93.2" 2020-10-15T17:01:49.843 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:01:49.843 INFO:teuthology.orchestra.run.smithi183.stdout: 2020-10-15T17:01:49.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:49.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:49.845 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:49.884 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:49.884 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:50.173 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T17:01:50.173 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000078", 2020-10-15T17:01:50.174 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.history", 2020-10-15T17:01:50.174 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.1", 2020-10-15T17:01:50.174 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000070", 2020-10-15T17:01:50.174 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000104", 2020-10-15T17:01:50.174 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000026", 2020-10-15T17:01:50.174 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000028", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.1", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000040", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.0", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-15:803109df-3f66-4c18-857a-2beb852ab289.4346.15:1", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000015", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000069", 2020-10-15T17:01:50.175 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000095", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000003", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000047", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000035", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000037", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000024", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000000", 2020-10-15T17:01:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000031", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:1", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.log.69fb12ff-ae85-466c-bb2d-3018777b1732.1", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.3", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:5", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:5", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000076", 2020-10-15T17:01:50.177 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000113", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000005", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000011", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000058", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000052", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000088", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000080", 2020-10-15T17:01:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000090", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000110", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000096", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.0", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000087", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000008", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:5", 2020-10-15T17:01:50.179 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000006", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000029", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000089", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000022", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-21:803109df-3f66-4c18-857a-2beb852ab289.4346.21:8", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "bilog.trim", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000016", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.2", 2020-10-15T17:01:50.180 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000041", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-46:803109df-3f66-4c18-857a-2beb852ab289.5150.3:8", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000018", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000092", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.2", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:7", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000014", 2020-10-15T17:01:50.181 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000112", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000007", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000021", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-22:803109df-3f66-4c18-857a-2beb852ab289.4346.22:8", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000064", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000071", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000074", 2020-10-15T17:01:50.182 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:7", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-14:803109df-3f66-4c18-857a-2beb852ab289.4346.14:1", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000081", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000009", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000121", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.1", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000125", 2020-10-15T17:01:50.183 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000082", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.1", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000105", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000059", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-49:803109df-3f66-4c18-857a-2beb852ab289.5352.1:5", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000077", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000032", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-25:803109df-3f66-4c18-857a-2beb852ab289.4346.25:8", 2020-10-15T17:01:50.184 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:8", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000053", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000091", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000065", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000083", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000010", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.2", 2020-10-15T17:01:50.185 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000045", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000002", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000116", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:3", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000034", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-37:803109df-3f66-4c18-857a-2beb852ab289.4346.37:10", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000101", 2020-10-15T17:01:50.186 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.1", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:1", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:8", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000079", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000049", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000117", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000044", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000066", 2020-10-15T17:01:50.187 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000068", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000085", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000073", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000038", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.3", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000118", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-20:803109df-3f66-4c18-857a-2beb852ab289.4346.20:8", 2020-10-15T17:01:50.188 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:8", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000036", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000103", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.3", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000119", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000098", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.log.0c649eb8-3e99-4d22-800d-adf2e2823b92.0", 2020-10-15T17:01:50.189 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status.shard.0", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000027", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "mdlog.sync-status", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000019", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000039", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:5", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000100", 2020-10-15T17:01:50.190 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:3", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000093", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000004", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000063", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000122", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.log.b84643c0-8302-4cfb-a70b-7af9afa8e893.1", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000057", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:3", 2020-10-15T17:01:50.191 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:3", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000054", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000114", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.0", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.shard.803109df-3f66-4c18-857a-2beb852ab289.3", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000012", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000084", 2020-10-15T17:01:50.192 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000043", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-12:803109df-3f66-4c18-857a-2beb852ab289.4346.12:1", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000111", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000048", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-10:803109df-3f66-4c18-857a-2beb852ab289.4346.10:1", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000020", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000099", 2020-10-15T17:01:50.193 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000056", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000072", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000062", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000115", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000033", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000025", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000123", 2020-10-15T17:01:50.194 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-36:803109df-3f66-4c18-857a-2beb852ab289.4346.36:10", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000108", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000094", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000060", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000109", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000013", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-13:803109df-3f66-4c18-857a-2beb852ab289.4346.13:1", 2020-10-15T17:01:50.195 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-18:803109df-3f66-4c18-857a-2beb852ab289.4346.18:3", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000051", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000106", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.0", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000017", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000023", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:1", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000107", 2020-10-15T17:01:50.196 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-26:803109df-3f66-4c18-857a-2beb852ab289.4346.26:8", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000046", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.full-sync.index.2", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-24:803109df-3f66-4c18-857a-2beb852ab289.4346.24:8", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "data_log.2", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000067", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-48:803109df-3f66-4c18-857a-2beb852ab289.5150.5:8", 2020-10-15T17:01:50.197 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-19:803109df-3f66-4c18-857a-2beb852ab289.4346.19:8", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000086", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-23:803109df-3f66-4c18-857a-2beb852ab289.4346.23:8", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-47:803109df-3f66-4c18-857a-2beb852ab289.5150.4:8", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-17:803109df-3f66-4c18-857a-2beb852ab289.4346.17:3", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000050", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000102", 2020-10-15T17:01:50.198 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-11:803109df-3f66-4c18-857a-2beb852ab289.4346.11:7", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000124", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000001", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000126", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "meta.log.0c649eb8-3e99-4d22-800d-adf2e2823b92.1", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000055", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "bucket.sync-status.803109df-3f66-4c18-857a-2beb852ab289:iybjjr-16:803109df-3f66-4c18-857a-2beb852ab289.4346.16:1", 2020-10-15T17:01:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000061", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000097", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "data.full-sync.index.803109df-3f66-4c18-857a-2beb852ab289.3", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "datalog.sync-status.803109df-3f66-4c18-857a-2beb852ab289", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000042", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000075", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000120", 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout: "obj_delete_at_hint.0000000030" 2020-10-15T17:01:50.200 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T17:01:50.201 INFO:teuthology.orchestra.run.smithi038.stdout: 2020-10-15T17:01:50.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:50.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:50.263 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:50.303 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:50.303 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:50.567 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:01:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-10-15T17:01:50.567 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.602422Z" 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000008871", 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.605096Z" 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:50.568 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003029", 2020-10-15T17:01:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.607734Z" 2020-10-15T17:01:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:01:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:01:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-15T17:01:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.610423Z" 2020-10-15T17:01:50.569 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:01:50.570 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:01:50.570 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-15T17:01:50.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:50.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:50.572 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:50.890 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:01:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:01:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:01:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:01:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:01:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 2547530715017092207 2020-10-15T17:01:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T17:01:50.892 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:13.436770Z" 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:50.893 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006495", 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:50.894 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:57:27.781546Z" 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:50.895 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.664298Z" 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:50.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:50.897 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T17:01:50.898 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T17:01:50.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:50.899 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:01:50.900 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702, shard=1 master=00000000000000000000:00000000000000008871 target=00000000000000000000:00000000000000006495, shard=2 master=00000000000000000000:00000000000000003029 target=00000000000000000000:00000000000000000646, shard=3 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374 2020-10-15T17:01:52.398 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T17:01:52.393+0000 7ffb6d7fa700 -1 virtual int RGWDataChangesFIFO::push(int, ceph::real_time, const string&, ceph::buffer::v15_2_0::list&&): unable to push to FIFO: data_log.2: (22) Invalid argument 2020-10-15T17:01:52.398 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T17:01:52.393+0000 7ffb6d7fa700 -1 ERROR: failed writing data log (info.bucket=:iybjjr-56[803109df-3f66-4c18-857a-2beb852ab289.5520.7]), shard_id=2) 2020-10-15T17:01:52.413 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T17:01:52.409+0000 7ffb68ff1700 -1 virtual int RGWDataChangesFIFO::push(int, ceph::real_time, const string&, ceph::buffer::v15_2_0::list&&): unable to push to FIFO: data_log.3: (22) Invalid argument 2020-10-15T17:01:52.414 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T17:01:52.409+0000 7ffb68ff1700 -1 ERROR: failed writing data log (info.bucket=:iybjjr-58[803109df-3f66-4c18-857a-2beb852ab289.5520.9]), shard_id=3) 2020-10-15T17:01:55.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:55.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:01:55.902 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:01:56.231 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:01:56.232 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:01:56.232 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:01:56.232 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:01:56.232 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:01:56.233 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 2547530715017092207 2020-10-15T17:01:56.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:56.233 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:01:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:01:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T17:01:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:13.436770Z" 2020-10-15T17:01:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:01:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006495", 2020-10-15T17:01:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:57:27.781546Z" 2020-10-15T17:01:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:01:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T17:01:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.664298Z" 2020-10-15T17:01:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:01:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:01:56.244 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:01:56.244 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:01:56.244 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T17:01:56.245 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:01:56.245 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:01:56.245 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:01:56.245 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T17:01:56.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:56.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:56.246 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:01:56.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:01:56.247 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:01:56.247 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T17:01:56.247 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T17:01:56.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:01:56.248 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:01:56.249 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702, shard=1 master=00000000000000000000:00000000000000008871 target=00000000000000000000:00000000000000006495, shard=2 master=00000000000000000000:00000000000000003029 target=00000000000000000000:00000000000000000646, shard=3 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374 2020-10-15T17:02:01.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:01.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:01.252 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:01.290 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:01.291 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:01.579 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:01.579 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 2547530715017092207 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:01.581 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:54:13.436770Z" 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:01.582 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006495", 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:57:27.781546Z" 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:01.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000000646", 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:01.584 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:51:43.664298Z" 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:01.585 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:46:08.169845Z" 2020-10-15T17:02:01.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:01.587 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:01.588 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702, shard=1 master=00000000000000000000:00000000000000008871 target=00000000000000000000:00000000000000006495, shard=2 master=00000000000000000000:00000000000000003029 target=00000000000000000000:00000000000000000646, shard=3 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374 2020-10-15T17:02:06.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:06.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:06.591 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:06.933 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:06.933 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 2547530715017092207 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:06.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:06.935 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:06.936 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.602422Z" 2020-10-15T17:02:06.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:06.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:06.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:06.936 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:06.936 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:06.937 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:06.937 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000008871", 2020-10-15T17:02:06.937 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:06.937 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:06.937 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:06.937 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.605096Z" 2020-10-15T17:02:06.938 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:06.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:06.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:06.938 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:06.938 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:06.938 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:06.939 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003029", 2020-10-15T17:02:06.939 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:06.939 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:06.939 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:06.939 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.607734Z" 2020-10-15T17:02:06.939 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:06.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:06.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:06.940 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:06.940 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:06.940 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:06.940 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.610423Z" 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:06.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T17:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T17:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:06.943 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-15T17:02:06.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:06.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:06.945 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:06.984 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:06.984 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:07.250 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:02:07.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.250 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.602422Z" 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000008871", 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.605096Z" 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003029", 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.607734Z" 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.610423Z" 2020-10-15T17:02:07.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.253 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:02:07.253 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-15T17:02:07.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:07.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:07.255 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:07.339 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:07.339 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:07.613 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:02:07.613 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:02:07.613 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:02:07.613 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 1541412607139480694 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:02:07.614 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:48.602422Z" 2020-10-15T17:02:07.615 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000008871", 2020-10-15T17:02:07.616 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:48.605096Z" 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:02:07.617 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003029", 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:48.607734Z" 2020-10-15T17:02:07.618 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-15T17:02:07.619 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:48.610423Z" 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:02:07.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.621 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:02:07.621 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T17:02:07.621 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T17:02:07.621 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.621 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:02:07.622 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-15T17:02:07.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:07.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:07.623 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:07.702 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:07.703 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:07.964 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:02:07.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.965 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-10-15T17:02:07.965 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.602422Z" 2020-10-15T17:02:07.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.966 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000008871", 2020-10-15T17:02:07.966 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.605096Z" 2020-10-15T17:02:07.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.966 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003029", 2020-10-15T17:02:07.966 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.607734Z" 2020-10-15T17:02:07.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:07.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:07.967 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-15T17:02:07.967 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:48.610423Z" 2020-10-15T17:02:07.967 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:07.967 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:02:07.968 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-15T17:02:07.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:07.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:07.969 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:08.012 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:08.012 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:08.383 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:08.383 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:08.383 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:08.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 16587652486901249284 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:08.384 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.602422Z" 2020-10-15T17:02:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000008871", 2020-10-15T17:02:08.386 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.605096Z" 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.387 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003029", 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:08.388 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.607734Z" 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-15T17:02:08.389 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:48.610423Z" 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:08.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.391 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:08.391 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T17:02:08.391 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T17:02:08.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.391 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:08.392 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-15T17:02:08.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:08.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:08.394 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:08.479 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:08.479 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:08.772 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T17:02:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-10-15T17:02:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.405185Z" 2020-10-15T17:02:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006886", 2020-10-15T17:02:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.410703Z" 2020-10-15T17:02:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.413438Z" 2020-10-15T17:02:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003566", 2020-10-15T17:02:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.401715Z" 2020-10-15T17:02:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:08.775 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T17:02:08.776 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-15T17:02:08.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:08.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:08.778 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:08.822 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:08.822 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:09.083 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:02:09.083 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:02:09.084 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:02:09.084 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:02:09.084 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:02:09.084 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 6408475988550126921 2020-10-15T17:02:09.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.084 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.085 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.086 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.086 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.405185Z" 2020-10-15T17:02:09.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.086 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006886", 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.087 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.410703Z" 2020-10-15T17:02:09.088 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.088 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:02:09.088 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.088 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.413438Z" 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003566", 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.090 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.401715Z" 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:02:09.091 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T17:02:09.092 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T17:02:09.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.092 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:02:09.093 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-15T17:02:09.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:09.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:09.094 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:09.133 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:09.134 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:09.434 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2020-10-15T17:02:09.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:09.435 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-10-15T17:02:09.435 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.405185Z" 2020-10-15T17:02:09.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:09.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:09.435 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006886", 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.410703Z" 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.413438Z" 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:09.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:09.437 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003566", 2020-10-15T17:02:09.437 INFO:teuthology.orchestra.run.smithi038.stdout: "last_update": "2020-10-15T17:01:52.401715Z" 2020-10-15T17:02:09.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:09.437 INFO:teuthology.orchestra.run.smithi038.stdout:] 2020-10-15T17:02:09.438 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-15T17:02:09.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:09.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:09.439 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:09.481 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:09.481 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:09.753 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:02:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:02:09.753 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:02:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:02:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:02:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 5492724315337703495 2020-10-15T17:02:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:02:09.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.755 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.405185Z" 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.756 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006886", 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.410703Z" 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:02:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:09.758 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.413438Z" 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:09.759 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003566", 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:09.760 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:52.401715Z" 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T17:02:09.761 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T17:02:09.762 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:09.762 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:02:09.763 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-15T17:02:09.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:09.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:09.764 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:09.843 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:09.843 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:10.104 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:02:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-10-15T17:02:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.381822Z" 2020-10-15T17:02:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006619", 2020-10-15T17:02:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.384051Z" 2020-10-15T17:02:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.386271Z" 2020-10-15T17:02:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-10-15T17:02:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.379617Z" 2020-10-15T17:02:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.108 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:02:10.109 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-15T17:02:10.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:10.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:10.111 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:10.192 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:10.192 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:10.445 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:02:10.445 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:02:10.445 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: "instance_id": 4508126650953487876 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:10.447 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:49.381822Z" 2020-10-15T17:02:10.448 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.448 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:02:10.448 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:10.448 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006619", 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:49.384051Z" 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.450 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:02:10.450 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:10.450 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:10.450 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:10.450 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:49.386271Z" 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.451 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "incremental-sync", 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:10.452 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:01:49.379617Z" 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 0, 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 0 2020-10-15T17:02:10.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.454 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:02:10.455 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-15T17:02:10.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:10.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:10.456 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:10.539 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:10.540 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:10.802 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:02:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-10-15T17:02:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.381822Z" 2020-10-15T17:02:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000006619", 2020-10-15T17:02:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.384051Z" 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.386271Z" 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-10-15T17:02:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:01:49.379617Z" 2020-10-15T17:02:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:10.805 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:02:10.806 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-15T17:02:10.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:10.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:10.807 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:10.850 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:10.851 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:11.135 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:11.135 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:11.136 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:11.136 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:11.136 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:11.136 INFO:teuthology.orchestra.run.smithi038.stdout: "instance_id": 11637474988163193673 2020-10-15T17:02:11.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:11.136 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000005023", 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:11.138 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:11.138 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:49.381822Z" 2020-10-15T17:02:11.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:11.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:11.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:11.138 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:11.139 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:11.139 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:11.139 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000006619", 2020-10-15T17:02:11.139 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:11.139 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:11.139 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:49.384051Z" 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:11.140 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000001838", 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:49.386271Z" 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:11.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:11.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:11.142 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:11.142 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:11.142 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "incremental-sync", 2020-10-15T17:02:11.142 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-10-15T17:02:11.142 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:49.379617Z" 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:11.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:11.144 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:11.144 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 0, 2020-10-15T17:02:11.144 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 0 2020-10-15T17:02:11.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:11.144 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:11.145 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-15T17:02:11.146 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-10-15T17:02:11.147 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-59 2020-10-15T17:02:11.195 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-60 2020-10-15T17:02:11.281 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-61 2020-10-15T17:02:11.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:11.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:11.919 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:11.967 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:11.967 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:12.216 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T17:02:12.253 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:02:12.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.254 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:02:12.254 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:02:12.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.254 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602781331.321841_457.1", 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:02:11.321841Z" 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.256 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:02:12.256 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:02:12.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.256 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:02:12.257 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T17:02:12.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:12.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:12.258 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:12.298 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:12.298 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:12.536 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:12.536 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:12.536 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:12.536 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:12.536 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:12.537 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:12.537 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:12.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:12.537 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:12.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:12.537 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:12.539 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781331.321841_457.1", 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:02:11.321841Z", 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:12.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:12.541 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:12.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:12.543 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:12.543 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:12.543 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:12.543 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:12.543 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:12.543 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:12.544 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:12.544 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:12.544 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:12.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:12.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:12.545 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:12.547 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T17:02:12.547 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T17:02:12.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:12.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:12.548 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:12.591 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:12.591 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:12.854 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:02:12.854 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:02:12.854 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:02:12.854 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:02:12.854 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:02:12.855 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:12.856 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:02:12.857 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602781331.321841_457.1", 2020-10-15T17:02:12.858 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:12.858 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:12.858 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:12.858 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:02:11.321841Z", 2020-10-15T17:02:12.863 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:02:12.863 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:02:12.865 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:02:12.866 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:02:12.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:02:12.868 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:02:12.869 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T17:02:12.869 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T17:02:12.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:12.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:12.871 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:13.181 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:13.182 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:13.182 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:13.182 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:13.182 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:13.183 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:13.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:13.186 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:13.186 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:13.186 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:13.186 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:13.186 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:13.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:13.187 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:13.189 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:13.190 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:13.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:13.191 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:13.192 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:18.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:18.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:18.194 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:18.485 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:18.495 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:18.496 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:23.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:23.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:23.499 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:23.539 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:23.539 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:23.914 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:23.915 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:23.915 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:23.915 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:23.915 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:23.915 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:23.915 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:23.916 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:23.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:23.918 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:23.919 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:23.920 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:23.921 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:23.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:23.923 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:23.924 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:28.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:28.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:28.927 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:28.968 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:28.968 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:29.241 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:29.250 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:29.251 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:34.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:34.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:34.253 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:34.290 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:34.290 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:34.559 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:34.640 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:34.641 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:39.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:39.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:39.644 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:39.683 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:39.683 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:40.056 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:40.057 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:40.057 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:40.057 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:40.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:40.057 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:40.057 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:40.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:40.058 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:40.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:40.058 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:40.058 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:40.059 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:40.060 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:40.061 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:40.061 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:40.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:40.061 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:40.061 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:40.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:40.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:40.064 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:40.064 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:40.064 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:40.065 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:40.065 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:40.065 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:40.065 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:40.065 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:40.066 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:40.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:40.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:40.066 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:40.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:40.066 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:40.067 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:40.067 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:40.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:40.067 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:40.068 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:45.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:45.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:45.071 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:45.111 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:45.111 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:45.381 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:45.391 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:45.391 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:50.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:50.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:50.393 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:50.433 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:50.434 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:50.805 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:50.806 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:50.806 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:50.806 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:50.806 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:50.806 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:50.807 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:50.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:50.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:50.811 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:50.812 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:50.813 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:50.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:50.815 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:50.816 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:02:55.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:55.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:02:55.818 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:02:55.860 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:02:55.861 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:02:56.233 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:02:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:02:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:02:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:02:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:02:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:02:56.234 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:02:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:02:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:56.235 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:56.236 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:02:56.237 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:56.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:02:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:56.239 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:02:56.240 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:02:56.241 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:02:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:02:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:02:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:02:56.242 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:02:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:02:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:02:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:02:56.243 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:02:56.244 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:01.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:01.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:01.246 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:01.553 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:01.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:01.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:01.559 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:01.560 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:01.561 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:01.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:01.562 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:01.563 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:06.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:06.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:06.566 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:06.608 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:06.608 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:06.876 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:06.877 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:06.878 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:06.879 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:06.879 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:06.879 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:06.879 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:06.879 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:06.879 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:06.880 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:06.881 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:06.881 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:06.881 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:06.881 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:06.881 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:06.881 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:06.882 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:06.884 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:06.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:06.886 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:06.886 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:06.886 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:06.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:06.886 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:06.887 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:11.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:11.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:11.890 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:11.932 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:11.932 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:12.303 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:12.303 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:12.303 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:12.303 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:12.304 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:12.305 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:12.306 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:12.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:12.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:12.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:12.306 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:12.306 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:12.307 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:12.308 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:12.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:12.310 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:12.311 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:12.312 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:12.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:12.312 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:12.313 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:17.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:17.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:17.315 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:17.355 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:17.355 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:17.728 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:17.729 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:17.729 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:17.729 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:17.729 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:17.729 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:17.729 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:17.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:17.730 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:17.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:17.734 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:17.735 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:17.736 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:17.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:17.738 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:17.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:17.738 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:17.738 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:17.738 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:17.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:17.739 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:17.740 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:22.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:22.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:22.741 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:22.784 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:22.784 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:23.161 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:23.162 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:23.162 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:23.162 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:23.162 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:23.162 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:23.163 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:23.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:23.163 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:23.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:23.163 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:23.163 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:23.164 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:23.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:23.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:23.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:23.165 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:23.165 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:23.165 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:23.166 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:23.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:23.169 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:23.170 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:23.171 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:23.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:23.171 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:23.172 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:28.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:28.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:28.174 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:28.216 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:28.216 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:28.592 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:28.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:28.594 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:28.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:28.594 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:28.594 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:28.594 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:28.594 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:28.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:28.596 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:28.597 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:28.597 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:28.597 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:28.597 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:28.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:28.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:28.598 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:28.598 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:28.598 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:28.598 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:28.599 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:28.599 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:28.599 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:28.599 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:28.599 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:28.600 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:28.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:28.602 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:28.603 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:33.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:33.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:33.606 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:33.647 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:33.917 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:33.918 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:33.918 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:33.918 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:33.918 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:33.918 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:33.918 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:33.919 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:33.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:33.921 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:33.922 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:33.923 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:33.924 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:33.926 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:33.927 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:38.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:38.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:38.930 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:38.972 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:38.973 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:39.345 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:39.355 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:39.356 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:44.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:44.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:44.358 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:44.400 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:44.400 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:44.678 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:44.688 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:44.688 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:49.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:49.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:49.691 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:49.732 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:49.732 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:49.997 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:49.998 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:49.998 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:49.998 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:49.998 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:49.998 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:49.998 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:49.999 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:50.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:50.003 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:50.004 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:50.005 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:50.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:50.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:50.006 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:50.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:50.006 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:50.007 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:50.007 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:50.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:50.007 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:50.008 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:03:55.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:55.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:03:55.011 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:03:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:03:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:03:55.425 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:03:55.425 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:55.426 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:55.427 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:55.428 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:55.429 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:55.430 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:55.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:03:55.432 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:03:55.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:03:55.434 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:03:55.434 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:03:55.434 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:03:55.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:03:55.434 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:03:55.435 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:00.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:00.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:00.438 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:00.480 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:00.481 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:00.854 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:00.864 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:00.865 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:05.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:05.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:05.868 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:05.912 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:05.913 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:06.283 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:06.283 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:06.283 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:06.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:06.286 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:06.287 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:06.288 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:06.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:06.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:06.288 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:06.288 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:06.288 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:06.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:06.290 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:06.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:06.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:06.292 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:06.292 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:06.292 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:06.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:06.292 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:06.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:11.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:11.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:11.296 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:11.336 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:11.337 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:11.608 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:11.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:11.610 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:11.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:11.610 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:11.610 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:11.610 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:11.610 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:11.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:11.612 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:11.613 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:11.614 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:11.615 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:11.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:11.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:11.618 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:11.618 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:16.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:16.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:16.621 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:16.663 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:16.664 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:16.934 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:16.934 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:16.934 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:16.934 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:16.934 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:16.935 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:16.936 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:16.937 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:16.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:16.939 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:16.940 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:16.941 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:16.942 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:16.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:16.943 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:16.944 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:21.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:21.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:21.946 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:21.987 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:21.987 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:22.255 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:22.264 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:22.265 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:27.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:27.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:27.268 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:27.675 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:27.675 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:27.675 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:27.675 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:27.676 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:27.676 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:27.676 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:27.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:27.676 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:27.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:27.677 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:27.679 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:27.680 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:27.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:27.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:27.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:27.680 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:27.680 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:27.681 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:27.682 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:27.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:27.684 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:27.685 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:32.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:32.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:32.688 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:32.731 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:32.732 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:32.999 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:32.999 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:32.999 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:32.999 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:32.999 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:33.000 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:33.000 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:33.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:33.001 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:33.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:33.001 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:33.001 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:33.001 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:33.001 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:33.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:33.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:33.003 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:33.003 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:33.003 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:33.003 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:33.003 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:33.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:33.006 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:33.007 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:33.008 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:33.008 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:33.008 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:33.008 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:33.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:33.008 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:33.009 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:33.009 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:33.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:33.009 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:33.010 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:38.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:38.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:38.013 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:38.069 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:38.070 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:38.338 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:38.353 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:38.354 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:43.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:43.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:43.357 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:43.399 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:43.400 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:43.768 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:43.768 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:43.768 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:43.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:43.770 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:43.771 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:43.772 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:43.773 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:43.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:43.775 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:43.776 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:43.777 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:43.778 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:48.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:48.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:48.781 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:48.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:48.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:49.196 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:49.196 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:49.196 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:49.196 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:49.196 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:49.197 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:49.198 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:49.199 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:49.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:49.201 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:49.202 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:49.203 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:49.203 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:49.203 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:49.203 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:49.203 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:49.203 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:49.204 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:49.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:49.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:49.204 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:49.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:49.204 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:49.205 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:49.205 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:49.205 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:49.205 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:49.206 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:54.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:54.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:54.208 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:54.250 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:54.251 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:04:54.618 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:04:54.619 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:04:54.619 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:04:54.619 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:04:54.619 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:04:54.620 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:04:54.621 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:54.622 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:54.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:54.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:54.624 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:04:54.624 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:54.624 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:54.624 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:54.624 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:54.625 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:54.625 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:54.625 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:54.625 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:54.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:54.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:04:54.626 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:04:54.627 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:04:54.628 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:04:54.628 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:04:54.628 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:04:54.628 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:04:54.629 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:04:59.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:59.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:04:59.632 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:04:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:04:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:00.044 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:00.044 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:00.044 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:00.044 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:00.044 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:00.045 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:00.046 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:00.047 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:00.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:00.049 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:00.050 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:00.051 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:00.052 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:00.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:00.053 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:05.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:05.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:05.056 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:05.098 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:05.099 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:05.466 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:05.467 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:05.467 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:05.467 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:05.467 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:05.467 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:05.467 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:05.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:05.468 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:05.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:05.468 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:05.468 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:05.469 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:05.470 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:05.471 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:05.471 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:05.471 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:05.471 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:05.471 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:05.471 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:05.472 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:05.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:05.475 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:05.476 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:05.476 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:05.476 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:05.476 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:05.477 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:10.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:10.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:10.480 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:10.523 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:10.523 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:10.897 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:10.898 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:10.898 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:10.898 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:10.898 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:10.899 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:10.900 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:10.901 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:10.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:10.903 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:10.904 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:10.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:10.906 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:10.907 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:10.908 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:15.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:15.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:15.911 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:15.951 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:15.951 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:16.329 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:16.338 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:16.339 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:21.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:21.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:21.341 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:21.385 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:21.385 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:21.753 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:21.771 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:21.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:26.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:26.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:26.775 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:26.818 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:26.819 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:27.192 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:27.193 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:27.193 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:27.193 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:27.193 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:27.194 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:27.194 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:27.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:27.194 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:27.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:27.194 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:27.195 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:27.195 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:27.195 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:27.195 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:27.195 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:27.195 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:27.196 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:27.196 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:27.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:27.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:27.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:27.196 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:27.197 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:27.197 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:27.197 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:27.197 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:27.197 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:27.198 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:27.198 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:27.198 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:27.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:27.199 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:27.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:27.201 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:27.202 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:27.202 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:27.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:27.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:27.202 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:27.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:27.203 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:27.203 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:27.203 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:27.203 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:27.203 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:27.204 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:32.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:32.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:32.208 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:32.248 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:32.249 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:32.602 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:32.603 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:32.603 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:32.603 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:32.603 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:32.603 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:32.603 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:32.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:32.604 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:32.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:32.604 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:32.604 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:32.604 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:32.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:32.606 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:32.607 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:32.608 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:32.609 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:32.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:32.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:32.612 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:32.613 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:37.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:37.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:37.615 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:37.657 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:38.033 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:38.042 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:38.043 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:43.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:43.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:43.045 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:43.087 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:43.088 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:43.461 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:43.462 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:43.462 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:43.462 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:43.462 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:43.462 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:43.463 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:43.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:43.465 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:43.466 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:43.467 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:43.468 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:43.469 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:43.469 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:43.469 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:43.469 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:43.469 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:43.470 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:43.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:43.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:43.470 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:43.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:43.470 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:43.471 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:43.471 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:43.471 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:43.471 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:43.472 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:48.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:48.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:48.475 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:48.517 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:48.518 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:48.884 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:48.886 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:48.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:48.886 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:48.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:48.886 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:48.886 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:48.888 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:48.889 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:48.889 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:48.889 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:48.889 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:48.889 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:48.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:48.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:48.890 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:48.890 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:48.890 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:48.890 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:48.890 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:48.891 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:48.891 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:48.891 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:48.891 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:48.891 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:48.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:48.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:48.892 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:48.892 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:48.892 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:48.892 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:48.893 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:48.893 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:48.893 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:48.893 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:48.893 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:48.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:48.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:48.894 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:48.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:48.894 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:48.894 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:48.895 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:48.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:48.895 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:48.896 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:53.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:53.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:53.899 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:53.938 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:53.939 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:54.302 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:54.302 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:54.302 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:54.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:54.304 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:54.305 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:54.306 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:54.307 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:54.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:54.309 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:54.310 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:54.311 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:54.311 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:54.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:54.311 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:54.312 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:05:59.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:59.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:05:59.315 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:05:59.357 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:05:59.357 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:05:59.731 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:05:59.731 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:05:59.731 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:05:59.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:59.733 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:05:59.733 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:59.733 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:59.733 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:05:59.733 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:59.733 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:59.734 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:59.735 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:05:59.736 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:05:59.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:59.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:59.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:59.736 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:05:59.736 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:59.737 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:05:59.738 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:05:59.739 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:05:59.739 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:05:59.739 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:05:59.739 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:05:59.739 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:05:59.739 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:05:59.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:05:59.741 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:05:59.742 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:04.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:04.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:04.744 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:04.787 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:04.788 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:05.154 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:05.154 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:05.154 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:05.156 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:05.157 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:05.158 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:05.159 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:05.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:05.161 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:05.162 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:05.163 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:05.163 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:05.163 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:05.164 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:10.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:10.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:10.166 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:10.208 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:10.208 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:10.569 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:10.569 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:10.570 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:10.571 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:10.572 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:10.573 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:10.574 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:10.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:10.576 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:10.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:10.578 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:10.578 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:10.578 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:10.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:10.578 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:10.579 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:15.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:15.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:15.581 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:15.625 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:15.625 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:15.996 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:15.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:15.998 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:16.000 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:16.001 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:16.002 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:16.003 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:16.004 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:16.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:16.005 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:16.005 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:16.005 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:16.005 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:16.005 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:16.006 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:21.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:21.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:21.009 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:21.050 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:21.422 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:21.427 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:21.428 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:21.431 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:21.432 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:26.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:26.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:26.434 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:26.477 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:26.478 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:26.953 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:26.953 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:26.953 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:26.954 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:26.954 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:26.954 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:26.954 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:26.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:26.954 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:26.955 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:26.956 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:26.957 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:26.958 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:26.959 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:26.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:26.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:26.960 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:26.960 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:26.961 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:26.961 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:26.961 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:26.961 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:26.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:26.962 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:26.963 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:26.963 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:26.963 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:26.963 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:26.964 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:31.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:31.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:31.971 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:32.011 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:32.011 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:32.287 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:32.288 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:32.288 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:32.288 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:32.288 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:32.288 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:32.288 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:32.289 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:32.290 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:32.290 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:32.290 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:32.290 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:32.290 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:32.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:32.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:32.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:32.292 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:32.292 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:32.292 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:32.292 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:32.293 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:32.293 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:32.293 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:32.294 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:32.294 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:32.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:32.295 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:32.296 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:32.296 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:32.296 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:32.296 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:32.297 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:32.297 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:32.297 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:32.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:32.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:32.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:32.298 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:32.298 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:32.298 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:32.299 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:32.299 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:32.299 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:32.300 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:32.300 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:32.300 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:32.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:32.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:32.301 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:32.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:32.301 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:32.302 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:32.302 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:32.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:32.303 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:32.304 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:37.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:37.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:37.306 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:37.347 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:37.347 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:37.715 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:37.725 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:37.726 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:42.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:42.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:42.728 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:42.772 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:42.772 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:43.144 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:43.144 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:43.144 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:43.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:43.146 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:43.147 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:43.148 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:43.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:43.151 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:43.152 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:43.153 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:43.153 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:43.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:43.153 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:43.154 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:48.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:48.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:48.157 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:48.556 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:48.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:48.558 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:48.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:48.558 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:48.558 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:48.558 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:48.558 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:48.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:48.560 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:48.561 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:48.562 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:48.563 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:48.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:48.565 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:48.567 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:53.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:53.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:53.570 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:53.610 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:53.610 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:53.981 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:53.981 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:53.981 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:53.981 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:53.982 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:53.983 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:53.984 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:53.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:53.986 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:53.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:53.988 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:53.989 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:53.990 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:53.990 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:53.990 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:53.990 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:53.991 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:06:58.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:58.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:06:58.993 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:06:59.036 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:06:59.036 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:06:59.406 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:06:59.407 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:06:59.407 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:06:59.407 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:06:59.407 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:06:59.407 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:59.408 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:06:59.410 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:59.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:59.412 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:06:59.413 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:06:59.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:06:59.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:06:59.416 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:06:59.417 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:04.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:04.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:04.419 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:04.461 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:04.461 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:04.837 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:04.838 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:04.838 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:04.838 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:04.838 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:04.839 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:04.840 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:04.841 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:04.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:04.843 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:04.844 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:04.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:04.847 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:04.848 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:09.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:09.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:09.850 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:09.892 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:09.893 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:10.266 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:10.278 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:10.279 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:15.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:15.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:15.282 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:15.696 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:15.696 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:15.697 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:15.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:15.700 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:15.701 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:15.702 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:15.703 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:15.704 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:15.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:15.705 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:15.705 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:15.705 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:15.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:15.705 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:15.706 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:20.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:20.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:20.709 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:20.751 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:20.751 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:21.121 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:21.122 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:21.122 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:21.122 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:21.122 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:21.122 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:21.123 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:21.124 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:21.124 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:21.124 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:21.124 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:21.124 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:21.124 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:21.125 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:21.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:21.127 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:21.128 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:21.129 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:21.130 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:21.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:21.131 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:21.132 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:26.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:26.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:26.135 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:26.175 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:26.175 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:26.549 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:26.550 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:26.550 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:26.550 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:26.550 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:26.550 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:26.550 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:26.551 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:26.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:26.553 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:26.554 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:26.556 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:26.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:26.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:26.559 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:26.559 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:31.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:31.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:31.562 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:31.602 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:31.603 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:31.972 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:31.973 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:31.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:31.976 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:31.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:31.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:31.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:31.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:31.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:31.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781331.321841_457.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:36.984 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2020-10-15T17:07:36.985 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-62 2020-10-15T17:07:37.030 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-63 2020-10-15T17:07:37.103 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-64 2020-10-15T17:07:37.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:37.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:37.772 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:37.814 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:37.814 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:38.050 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T17:07:38.074 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:07:38.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.075 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:07:38.075 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:07:38.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.076 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602781657.152264_498.1", 2020-10-15T17:07:38.076 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:07:37.152264Z" 2020-10-15T17:07:38.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.077 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:07:38.077 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:07:38.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.078 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:07:38.078 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:07:38.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.078 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:07:38.079 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T17:07:38.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:38.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:38.081 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:38.124 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:38.124 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:38.385 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781657.152264_498.1", 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:07:37.152264Z", 2020-10-15T17:07:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:38.394 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:38.395 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T17:07:38.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T17:07:38.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:38.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:38.397 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:38.436 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:38.436 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:38.669 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:07:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:07:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:07:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:07:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:07:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:07:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:07:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:07:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602781657.152264_498.1", 2020-10-15T17:07:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:07:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:07:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:07:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:07:37.152264Z", 2020-10-15T17:07:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:07:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:07:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:07:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:07:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:07:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:07:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:07:38.679 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:07:38.680 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T17:07:38.680 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T17:07:38.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:38.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:38.682 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:38.723 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:38.724 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:39.093 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:39.103 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:39.104 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:44.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:44.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:44.105 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:44.623 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:44.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:44.625 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:44.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:44.625 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:44.625 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:44.625 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:44.625 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:44.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:44.627 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:44.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:44.629 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:44.629 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:44.629 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:44.629 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:44.629 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:44.629 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:44.630 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:44.631 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:44.632 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:44.633 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:44.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:44.633 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:44.634 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:49.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:49.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:49.637 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:49.680 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:49.948 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:49.954 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:49.955 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:49.956 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:49.957 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:49.958 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:49.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:49.958 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:49.959 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:07:54.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:54.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:07:54.962 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:07:55.004 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:07:55.004 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:07:55.368 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:07:55.368 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:07:55.369 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:07:55.369 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:07:55.369 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:07:55.369 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:07:55.369 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:55.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:07:55.370 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:55.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:55.372 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:07:55.373 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:55.374 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:07:55.375 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:55.376 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:07:55.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:07:55.377 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:07:55.377 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:07:55.377 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:07:55.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:07:55.377 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:07:55.378 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:00.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:00.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:00.381 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:00.424 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:00.424 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:00.798 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:00.808 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:00.809 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:05.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:05.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:05.811 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:06.328 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:06.328 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:06.329 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:06.329 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:06.329 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:06.329 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:06.329 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:06.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:06.330 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:06.331 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:06.331 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:06.331 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:06.331 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:06.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:06.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:06.332 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:06.333 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:06.335 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:06.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:06.337 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:06.338 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:11.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:11.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:11.341 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:11.382 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:11.382 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:11.654 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:11.655 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:11.655 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:11.655 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:11.655 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:11.655 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:11.656 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:11.658 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:11.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:11.660 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:11.661 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:11.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:11.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:11.664 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:11.664 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:16.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:16.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:16.667 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:16.707 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:16.708 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:17.078 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:17.087 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:17.088 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:22.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:22.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:22.090 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:22.132 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:22.132 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:22.396 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:22.397 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:22.397 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:22.397 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:22.397 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:22.397 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:22.397 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:22.398 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:22.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:22.400 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:22.401 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:22.402 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:22.402 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:22.402 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:22.402 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:22.402 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:22.403 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:22.403 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:22.403 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:22.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:22.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:22.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:22.404 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:22.404 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:22.404 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:22.404 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:22.404 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:22.407 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:22.408 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:27.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:27.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:27.410 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:27.451 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:27.451 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:27.825 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:27.834 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:27.835 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:32.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:32.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:32.837 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:33.250 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:33.254 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:33.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:33.256 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:33.257 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:33.258 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:33.259 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:33.260 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:33.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:33.260 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:33.260 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:38.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:38.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:38.263 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:38.303 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:38.303 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:38.666 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:38.675 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:38.676 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:43.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:43.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:43.678 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:43.720 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:43.721 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:44.096 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:44.096 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:44.096 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:44.097 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:44.097 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:44.097 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:44.097 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:44.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:44.097 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:44.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:44.098 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:44.098 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:44.098 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:44.098 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:44.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:44.100 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:44.100 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:44.100 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:44.100 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:44.100 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:44.100 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:44.101 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:44.102 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:44.102 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:44.102 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:44.103 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:44.103 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:44.103 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:44.103 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:44.103 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:44.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:44.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:44.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:44.104 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:44.104 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:44.104 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:44.104 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:44.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:44.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:44.106 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:44.106 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:44.106 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:44.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:44.107 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:44.108 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:49.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:49.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:49.111 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:49.152 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:49.153 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:49.528 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:49.529 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:49.529 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:49.529 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:49.529 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:49.529 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:49.529 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:49.530 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:49.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:49.532 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:49.533 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:49.534 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:49.535 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:49.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:49.537 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:49.538 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:54.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:54.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:54.541 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:08:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:08:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:08:54.962 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:08:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:08:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:08:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:08:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:08:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:08:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:08:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:08:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:08:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:08:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:08:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:08:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:08:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:08:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:08:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:08:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:08:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:08:54.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:08:54.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:08:59.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:59.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:08:59.985 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:00.026 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:00.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:00.491 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:00.510 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:00.511 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:05.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:05.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:05.514 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:05.555 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:05.923 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:05.932 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:05.933 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:10.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:10.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:10.936 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:11.348 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:11.348 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:11.348 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:11.348 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:11.349 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:11.350 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:11.350 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:11.350 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:11.350 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:11.350 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:11.350 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:11.351 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:11.351 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:11.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:11.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:11.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:11.351 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:11.352 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:11.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:11.355 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:11.357 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:11.357 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:11.357 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:11.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:11.357 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:11.358 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:16.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:16.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:16.361 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:16.402 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:16.402 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:16.773 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:16.774 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:16.774 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:16.774 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:16.774 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:16.774 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:16.775 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:16.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:16.775 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:16.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:16.775 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:16.776 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:16.777 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:16.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:16.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:16.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:16.777 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:16.778 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:16.778 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:16.778 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:16.778 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:16.779 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:16.779 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:16.779 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:16.779 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:16.780 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:16.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:16.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:16.780 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:16.781 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:16.781 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:16.781 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:16.781 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:16.781 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:16.781 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:16.782 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:16.782 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:16.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:16.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:16.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:16.782 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:16.783 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:16.784 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:16.785 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:16.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:16.785 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:16.786 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:21.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:21.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:21.792 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:21.834 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:21.835 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:22.211 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:22.212 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:22.212 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:22.212 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:22.212 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:22.212 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:22.213 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:22.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:22.215 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:22.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:22.217 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:22.218 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:22.219 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:22.219 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:22.219 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:22.219 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:22.219 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:22.219 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:22.220 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:22.222 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:22.223 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:27.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:27.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:27.225 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:27.269 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:27.625 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:27.626 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:27.626 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:27.626 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:27.627 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:27.627 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:27.627 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:27.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:27.628 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:27.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:27.628 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:27.628 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:27.629 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:27.629 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:27.629 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:27.630 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:27.630 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:27.630 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:27.630 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:27.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:27.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:27.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:27.631 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:27.632 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:27.632 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:27.632 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:27.632 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:27.633 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:27.633 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:27.633 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:27.633 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:27.634 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:27.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:27.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:27.634 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:27.635 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:27.635 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:27.635 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:27.635 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:27.636 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:27.636 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:27.636 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:27.637 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:27.637 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:27.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:27.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:27.637 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:27.638 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:27.638 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:27.638 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:27.638 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:27.638 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:27.639 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:27.639 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:27.639 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:27.639 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:27.639 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:27.640 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:27.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:27.640 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:27.640 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:27.641 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:27.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:27.641 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:27.643 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:32.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:32.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:32.645 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:32.686 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:32.686 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:33.054 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:33.055 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:33.055 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:33.055 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:33.055 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:33.055 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:33.056 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:33.057 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:33.057 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:33.057 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:33.057 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:33.057 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:33.057 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:33.058 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:33.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:33.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:33.058 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:33.058 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:33.058 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:33.059 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:33.059 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:33.059 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:33.059 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:33.059 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:33.059 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:33.060 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:33.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:33.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:33.062 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:33.062 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:33.062 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:33.062 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:33.062 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:33.063 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:33.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:33.064 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:33.064 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:33.064 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:33.064 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:33.064 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:33.065 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:38.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:38.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:38.068 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:38.111 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:38.111 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:38.484 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:38.493 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:38.494 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:43.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:43.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:43.497 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:43.540 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:43.541 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:43.905 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:43.915 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:43.916 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:48.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:48.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:48.918 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:48.960 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:48.960 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:49.225 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:49.225 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:49.225 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:49.227 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:49.228 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:49.229 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:49.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:49.232 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:49.234 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:49.234 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:49.234 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:49.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:49.234 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:49.235 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:54.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:54.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:54.378 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:54.422 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:54.422 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:09:54.794 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:09:54.794 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:09:54.795 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:09:54.795 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:09:54.795 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:09:54.795 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:09:54.795 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:54.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:09:54.796 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:54.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:54.798 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:09:54.799 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:54.800 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:09:54.801 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:54.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:09:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:09:54.804 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:09:59.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:59.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:09:59.807 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:09:59.847 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:09:59.848 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:00.123 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:00.123 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:00.124 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:00.124 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:00.124 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:00.125 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:00.125 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:00.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:00.126 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:00.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:00.126 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:00.126 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:00.127 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:00.127 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:00.127 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:00.128 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:00.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:00.128 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:00.128 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:00.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:00.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:00.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:00.130 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:00.130 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:00.130 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:00.130 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:00.131 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:00.131 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:00.131 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:00.132 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:00.132 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:00.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:00.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:00.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:00.133 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:00.133 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:00.134 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:00.134 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:00.134 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:00.134 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:00.136 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:00.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:00.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:00.136 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:00.137 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:00.137 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:00.137 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:00.138 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:00.138 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:00.138 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:00.138 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:00.139 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:00.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:00.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:00.140 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:00.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:00.140 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:00.140 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:00.141 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:00.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:00.141 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:00.143 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:05.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:05.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:05.145 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:05.187 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:05.187 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:05.556 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:05.556 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:05.557 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:05.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:05.559 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:05.561 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:05.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:05.563 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:05.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:05.565 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:05.565 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:05.565 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:05.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:05.565 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:05.566 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:10.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:10.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:10.569 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:10.610 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:10.610 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:10.973 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:10.973 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:10.975 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:10.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:10.977 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:10.978 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:10.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:10.980 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:10.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:10.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:10.982 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:10.982 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:10.982 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:10.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:10.982 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:10.983 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:15.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:15.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:15.986 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:16.028 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:16.400 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:16.407 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:16.409 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:16.410 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:21.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:21.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:21.412 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:21.454 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:21.454 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:21.833 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:21.842 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:21.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:21.843 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:21.844 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:26.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:26.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:26.846 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:26.887 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:26.887 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:27.257 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:27.258 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:27.258 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:27.258 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:27.258 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:27.258 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:27.258 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:27.259 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:27.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:27.261 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:27.262 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:27.263 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:27.264 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:27.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:27.266 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:27.267 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:32.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:32.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:32.270 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:32.689 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:32.690 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:32.690 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:32.690 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:32.690 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:32.690 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:32.691 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:32.693 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:32.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:32.695 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:32.696 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:32.697 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:32.698 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:32.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:32.699 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:32.700 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:37.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:37.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:37.704 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:37.746 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:37.747 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:38.117 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:38.127 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:38.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:43.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:43.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:43.130 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:43.172 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:43.173 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:43.541 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:43.541 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:43.542 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:43.542 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:43.543 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:43.543 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:43.543 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:43.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:43.545 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:43.546 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:43.546 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:43.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:43.547 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:43.547 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:43.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:43.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:43.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:43.548 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:43.549 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:43.549 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:43.549 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:43.550 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:43.550 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:43.550 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:43.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:43.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:43.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:43.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:43.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:43.552 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:43.553 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:43.553 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:43.555 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:43.555 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:43.555 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:43.556 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:43.556 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:43.556 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:43.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:43.559 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:43.559 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:43.559 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:43.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:43.559 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:43.560 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:48.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:48.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:48.563 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:48.603 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:48.603 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:48.966 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:48.976 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:48.977 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:53.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:53.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:53.980 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:54.021 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:54.022 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:10:54.390 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:10:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:10:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:10:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:10:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:10:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:10:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:54.392 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:54.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:54.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:54.394 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:10:54.394 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:54.394 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:54.394 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:10:54.394 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:54.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:54.396 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:10:54.397 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:10:54.398 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:10:54.398 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:10:54.398 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:10:54.398 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:10:54.398 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:10:54.398 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:10:54.399 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:10:54.400 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:10:54.400 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:10:54.400 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:10:54.401 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:10:59.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:59.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:10:59.655 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:10:59.699 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:10:59.699 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:00.069 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:00.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:00.080 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:00.081 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:05.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:05.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:05.083 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:05.126 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:05.126 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:05.501 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:05.501 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:05.502 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:05.502 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:05.502 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:05.503 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:05.503 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:05.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:05.503 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:05.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:05.504 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:05.504 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:05.505 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:05.505 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:05.505 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:05.505 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:05.506 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:05.506 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:05.506 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:05.507 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:05.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:05.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:05.508 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:05.508 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:05.508 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:05.508 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:05.509 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:05.509 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:05.509 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:05.510 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:05.510 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:05.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:05.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:05.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:05.511 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:05.511 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:05.512 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:05.512 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:05.512 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:05.513 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:05.513 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:05.513 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:05.514 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:05.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:05.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:05.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:05.515 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:05.515 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:05.516 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:05.516 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:05.516 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:05.517 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:05.517 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:05.517 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:05.517 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:05.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:05.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:05.518 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:05.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:05.519 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:05.519 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:05.520 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:05.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:05.520 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:05.522 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:10.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:10.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:10.524 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:10.563 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:10.564 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:10.937 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:10.937 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:10.938 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:10.938 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:10.938 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:10.938 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:10.938 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:10.953 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:10.955 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:15.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:15.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:15.958 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:15.999 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:15.999 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:16.367 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:16.377 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:21.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:21.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:21.379 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:21.419 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:21.420 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:21.798 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:21.808 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:21.809 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:26.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:26.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:26.811 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:26.852 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:26.852 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:27.224 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:27.225 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:27.225 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:27.225 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:27.225 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:27.225 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:27.226 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:27.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:27.226 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:27.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:27.226 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:27.226 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:27.227 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:27.228 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:27.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:27.230 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:27.231 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:27.232 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:27.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:27.233 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:27.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:27.233 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:27.233 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:27.233 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:27.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:27.234 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:27.235 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:32.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:32.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:32.237 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:32.279 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:32.280 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:32.649 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:32.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:32.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:32.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:32.653 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:32.653 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:32.653 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:32.653 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:32.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:32.655 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:32.656 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:32.657 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:32.657 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:32.657 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:32.657 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:32.657 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:32.657 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:32.658 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:32.658 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:32.659 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:32.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:32.660 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:32.661 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:37.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:37.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:37.664 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:37.706 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:37.706 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:38.080 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:38.099 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:38.101 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:43.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:43.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:43.104 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:43.147 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:43.148 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:43.521 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:43.522 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:43.522 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:43.522 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:43.522 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:43.522 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:43.522 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:43.523 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:43.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:43.525 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:43.526 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:43.527 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:43.528 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:43.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:43.530 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:43.531 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:48.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:48.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:48.534 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:48.580 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:48.580 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:48.956 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:48.957 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:48.957 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:48.957 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:48.957 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:48.957 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:48.958 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:48.964 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:48.966 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:48.967 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:53.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:53.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:53.969 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:54.012 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:54.012 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:54.382 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:54.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:54.384 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:54.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:54.384 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:54.384 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:54.384 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:54.384 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:54.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:54.386 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:54.387 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:54.388 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:54.389 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:54.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:54.391 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:54.392 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:11:59.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:59.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:11:59.396 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:11:59.436 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:11:59.436 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:11:59.810 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:11:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:11:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:11:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:11:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:11:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:11:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:11:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:11:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:11:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:11:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:11:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:11:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:11:59.819 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:11:59.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:04.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:04.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:04.824 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:04.868 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:04.868 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:05.338 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:05.348 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:05.349 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:10.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:10.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:10.351 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:10.394 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:10.395 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:10.761 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:10.770 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:10.771 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:15.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:15.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:15.774 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:15.814 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:15.815 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:16.176 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:16.176 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:16.177 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:16.178 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:16.179 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:16.180 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:16.181 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:16.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:16.183 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:16.184 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:16.185 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:16.185 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:16.185 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:16.185 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:16.186 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:21.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:21.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:21.188 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:21.236 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:21.236 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:21.594 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:21.594 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:21.596 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:21.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:21.598 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:21.599 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:21.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:21.601 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:21.602 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:21.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:21.603 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:21.603 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:21.603 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:21.603 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:21.603 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:21.604 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:26.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:26.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:26.606 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:26.647 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:26.648 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:27.019 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:27.020 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:27.020 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:27.020 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:27.020 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:27.020 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:27.021 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:27.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:27.021 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:27.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:27.021 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:27.022 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:27.023 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:27.023 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:27.023 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:27.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:27.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:27.023 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:27.024 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:27.025 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:27.026 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:27.026 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:27.026 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:27.026 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:27.026 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:27.026 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:27.027 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:27.028 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:27.029 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:27.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:27.029 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:27.029 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:27.029 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:27.029 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:27.030 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:27.031 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:32.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:32.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:32.033 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:32.442 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:32.442 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:32.442 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:32.442 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:32.443 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:32.444 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:32.445 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:32.446 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:32.447 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:32.448 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:32.449 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:32.450 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:32.451 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:32.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:32.451 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:32.452 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:37.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:37.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:37.455 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:37.497 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:37.497 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:37.856 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:37.865 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:37.866 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:42.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:42.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:42.869 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:42.911 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:42.912 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:43.283 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:43.286 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:43.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:43.288 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:43.289 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:43.290 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:43.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:43.292 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:43.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:48.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:48.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:48.295 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:48.706 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:48.707 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:48.707 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:48.707 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:48.707 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:48.707 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:48.708 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:48.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:48.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:48.710 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:48.710 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:48.710 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:48.710 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:48.710 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:48.711 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:48.712 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:48.713 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:48.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:48.716 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:48.717 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:53.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:53.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:53.720 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:54.132 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:54.141 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:54.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:12:59.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:59.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:12:59.145 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:12:59.188 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:12:59.188 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:12:59.556 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:12:59.557 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:12:59.557 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:12:59.557 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:12:59.557 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:12:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:12:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:12:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:12:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:12:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:12:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:12:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:12:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:12:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:12:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:12:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:12:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:12:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:12:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:12:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:12:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:12:59.568 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:12:59.569 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602781657.152264_498.1 target=1_1602781307.714634_418.1 2020-10-15T17:13:04.571 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2020-10-15T17:13:04.572 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-65 2020-10-15T17:13:04.619 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=iybjjr-66 2020-10-15T17:13:04.717 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=iybjjr-67 2020-10-15T17:13:08.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:08.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:08.076 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:08.127 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:08.127 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:08.374 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:08.372+0000 7f6e7f20e0c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:08.374 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000001.31.5, , , , , 00000000001.57.5] 2020-10-15T17:13:08.374 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:13.378 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:13.375+0000 7f6e7f20e0c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:13.378 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000001.31.5, , , , , 00000000001.57.5] 2020-10-15T17:13:13.378 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:18.382 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:18.379+0000 7f6e7f20e0c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:18.383 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000001.31.5, , , , , 00000000001.57.5] 2020-10-15T17:13:18.383 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:23.386 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:23.383+0000 7f6e7f20e0c0 1 bucket sync caught up with source: 2020-10-15T17:13:23.387 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:23.387 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:23.387 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:23.383+0000 7f6e7f20e0c0 0 bucket checkpoint complete 2020-10-15T17:13:23.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:23.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:23.422 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:23.469 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:23.469 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:23.710 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:23.709+0000 7faa196560c0 1 bucket sync caught up with source: 2020-10-15T17:13:23.710 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:23.711 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:23.711 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:23.709+0000 7faa196560c0 0 bucket checkpoint complete 2020-10-15T17:13:23.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:23.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:23.740 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:23.787 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:23.788 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:24.054 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:24.051+0000 7f69d5bea0c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:24.054 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000001.31.5, , , , , 00000000001.57.5] 2020-10-15T17:13:24.054 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:29.058 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:29.055+0000 7f69d5bea0c0 1 bucket sync caught up with source: 2020-10-15T17:13:29.058 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:29.058 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000010.41.16, , , , , 00000000001.57.5] 2020-10-15T17:13:29.059 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:29.055+0000 7f69d5bea0c0 0 bucket checkpoint complete 2020-10-15T17:13:29.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:29.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:29.167 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:29.212 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:29.212 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:29.458 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:29.457+0000 7f1f70e880c0 1 bucket sync caught up with source: 2020-10-15T17:13:29.458 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000001.36.5, , , , , ] 2020-10-15T17:13:29.458 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000001.36.5, , , , , ] 2020-10-15T17:13:29.458 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:29.457+0000 7f1f70e880c0 0 bucket checkpoint complete 2020-10-15T17:13:29.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:29.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:29.489 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:29.573 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:29.573 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:29.836 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:29.833+0000 7f699469f0c0 1 bucket sync caught up with source: 2020-10-15T17:13:29.836 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000001.36.5, , , , , ] 2020-10-15T17:13:29.837 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000001.36.5, , , , , ] 2020-10-15T17:13:29.837 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:29.833+0000 7f699469f0c0 0 bucket checkpoint complete 2020-10-15T17:13:29.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:29.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:29.867 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:29.909 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:29.910 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:30.157 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:30.157+0000 7f19fc0100c0 1 bucket sync caught up with source: 2020-10-15T17:13:30.158 INFO:teuthology.orchestra.run.smithi183.stderr: local status: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:30.158 INFO:teuthology.orchestra.run.smithi183.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:30.158 INFO:teuthology.orchestra.run.smithi183.stderr:2020-10-15T17:13:30.157+0000 7f19fc0100c0 0 bucket checkpoint complete 2020-10-15T17:13:30.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:30.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'iybjjr-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-15T17:13:30.186 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket iybjjr-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-15T17:13:30.239 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:13:30.239 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:13:30.505 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:30.503+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:30.505 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:13:30.506 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:35.509 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:35.507+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:35.509 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:13:35.510 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:40.513 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:40.511+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:40.513 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:13:40.514 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:45.517 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:45.515+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:45.517 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:13:45.518 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:50.521 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:50.519+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:50.521 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:13:50.522 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:13:55.526 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:13:55.519+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:13:55.526 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:13:55.527 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:00.527 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:00.523+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:00.528 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:00.528 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:05.531 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:05.526+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:05.532 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:05.532 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:10.535 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:10.530+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:10.535 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:10.535 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:15.536 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:15.530+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:15.537 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:15.537 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:20.540 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:20.534+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:20.540 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:20.541 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:25.545 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:25.538+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:25.545 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:25.545 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:30.548 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:30.542+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:30.549 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:30.549 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:35.552 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:35.546+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:35.553 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:35.553 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:40.557 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:40.554+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:40.558 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:40.558 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:45.560 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:45.554+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:45.561 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:45.561 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:50.564 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:50.558+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:50.565 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:50.565 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:14:55.568 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:14:55.562+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:14:55.569 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:14:55.569 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:00.573 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:00.565+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:15:00.573 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:15:00.573 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:05.576 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:05.569+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:15:05.576 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:15:05.577 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:10.580 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:10.573+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:15:10.580 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:15:10.580 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:15.583 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:15.577+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:15:15.584 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:15:15.584 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:20.586 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:20.581+0000 7f0c4d9480c0 1 waiting for incremental sync to catch up: 2020-10-15T17:15:20.587 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , , , , , , ] 2020-10-15T17:15:20.587 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:25.585+0000 7f0c4d9480c0 1 bucket sync caught up with source: 2020-10-15T17:15:25.591 INFO:teuthology.orchestra.run.smithi038.stderr: local status: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:25.591 INFO:teuthology.orchestra.run.smithi038.stderr: remote markers: [, , , , , 00000000001.33.5, , , , , ] 2020-10-15T17:15:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:2020-10-15T17:15:25.585+0000 7f0c4d9480c0 0 bucket checkpoint complete 2020-10-15T17:15:25.627 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-65 zones={test-zone2, test-zone1} 2020-10-15T17:15:25.653 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T17:15:25.707 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-10-15T17:15:25.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T17:15:25.716 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-10-15T17:15:25.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:25.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:26.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:26.502 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-65 zones={test-zone2, test-zone1} 2020-10-15T17:15:26.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-65 zones={test-zone3, test-zone1} 2020-10-15T17:15:26.515 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T17:15:26.562 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-10-15T17:15:26.563 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T17:15:26.570 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-10-15T17:15:26.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:26.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:27.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:27.325 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-65 zones={test-zone3, test-zone1} 2020-10-15T17:15:27.325 INFO:tasks.rgw_multi.zone_rados:comparing bucket=iybjjr-65 zones={test-zone3, test-zone2} 2020-10-15T17:15:27.336 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-15T17:15:27.382 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-10-15T17:15:27.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-15T17:15:27.391 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-10-15T17:15:27.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:27.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:27.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-10-15T17:15:28.170 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=iybjjr-65 zones={test-zone3, test-zone2} 2020-10-15T17:15:28.171 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-10-15T17:15:28.172 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-68 2020-10-15T17:15:28.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:28.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:28.308 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:28.347 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:28.348 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:28.404 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T17:15:28.401+0000 7f5697fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T17:15:28.405 INFO:tasks.rgw.c1.client.1.smithi183.stdout:2020-10-15T17:15:28.401+0000 7f5697fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T17:15:28.405 INFO:tasks.rgw.c2.client.0.smithi038.stdout:2020-10-15T17:15:28.397+0000 7ff0f1ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T17:15:28.405 INFO:tasks.rgw.c2.client.0.smithi038.stdout:2020-10-15T17:15:28.401+0000 7ff0eeff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T17:15:28.406 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T17:15:28.401+0000 7ffb767fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T17:15:28.406 INFO:tasks.rgw.c2.client.1.smithi038.stdout:2020-10-15T17:15:28.401+0000 7ffb6b7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-15T17:15:28.579 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T17:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602782128.213216_529.1", 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:15:28.213216Z" 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:15:28.607 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T17:15:28.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:28.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:28.609 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:28.650 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:28.651 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:28.902 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:28.903 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:28.903 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:28.903 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:28.903 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:28.903 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:28.904 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:28.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602782128.213216_529.1", 2020-10-15T17:15:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:15:28.213216Z", 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:28.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:28.908 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:28.909 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:28.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:28.911 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:28.913 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T17:15:28.913 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T17:15:28.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:28.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:28.914 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:28.951 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:28.952 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:29.205 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:15:29.206 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:15:29.206 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:15:29.206 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:15:29.206 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:15:29.206 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:29.206 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:15:29.207 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:29.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602782128.213216_529.1", 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:15:29.209 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:15:28.213216Z", 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:15:29.210 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:15:29.211 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:15:29.212 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:29.213 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:15:29.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:15:29.214 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:15:29.214 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:15:29.214 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:15:29.214 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:15:29.214 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:15:29.215 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T17:15:29.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T17:15:29.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:29.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:29.217 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:29.257 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:29.258 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:29.624 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:29.624 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:29.625 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:29.626 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:29.627 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:29.628 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:29.629 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:29.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:29.631 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:29.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:29.633 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:29.633 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:29.633 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:29.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:29.633 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:29.634 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:15:34.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:34.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:34.637 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:34.687 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:34.687 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:35.062 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:15:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:35.074 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:35.076 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:15:40.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:40.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:40.078 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:40.493 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:40.493 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:40.495 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:40.496 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:40.496 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:40.496 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:40.496 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:40.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:40.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:40.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:40.497 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:40.497 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:40.497 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:40.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:40.499 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:40.501 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:40.502 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:40.503 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:40.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:40.503 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:40.504 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:15:45.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:45.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:45.507 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:45.549 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:45.549 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:45.928 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:45.930 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:45.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:45.932 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:45.933 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:45.934 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:45.935 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:45.936 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:45.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:45.937 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:45.937 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:45.937 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:45.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:45.937 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:45.938 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:15:50.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:50.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:50.941 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:50.982 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:50.983 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:51.350 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:51.350 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:51.351 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:51.352 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:51.353 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:51.353 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:51.353 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:51.353 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:51.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:51.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:51.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:51.354 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:51.354 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:51.354 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:51.354 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:15:51.354 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:51.355 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:51.355 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:51.355 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:15:51.355 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:51.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:51.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:51.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:51.356 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:51.356 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:51.356 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:51.356 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:51.356 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:51.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:51.358 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:51.358 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:51.358 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:51.358 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:51.358 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:51.358 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:51.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:51.360 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:51.360 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:51.360 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:51.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:51.360 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:51.361 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:15:56.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:56.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:15:56.364 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:15:56.405 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:15:56.406 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:15:56.775 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:15:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:15:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:15:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:15:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:15:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:15:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:56.780 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:15:56.782 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:15:56.783 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:15:56.784 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:15:56.784 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:15:56.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:15:56.784 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:15:56.785 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:01.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:01.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:01.788 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:01.830 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:01.831 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:02.209 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:02.210 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:02.211 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:02.211 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:02.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:02.212 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:02.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:02.215 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:02.216 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:02.217 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:02.218 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:02.219 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:02.222 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:02.223 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:02.224 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:02.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:02.225 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:02.227 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:07.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:07.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:07.230 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:07.274 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:07.275 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:07.641 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:07.642 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:07.642 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:07.642 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:07.642 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:07.643 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:07.643 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:07.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:07.643 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:07.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:07.643 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:07.644 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:07.644 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:07.644 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:07.644 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:07.645 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:07.645 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:07.645 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:07.645 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:07.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:07.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:07.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:07.647 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:07.647 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:07.647 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:07.648 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:07.648 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:07.648 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:07.649 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:07.649 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:07.649 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:07.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:07.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:07.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:07.650 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:07.651 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:07.651 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:07.651 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:07.651 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:07.652 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:07.652 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:07.652 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:07.652 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:07.653 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:07.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:07.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:07.654 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:07.654 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:07.654 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:07.654 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:07.655 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:07.655 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:07.655 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:07.656 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:07.656 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:07.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:07.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:07.657 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:07.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:07.657 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:07.657 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:07.658 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:07.658 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:07.658 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:07.660 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:12.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:12.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:12.662 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:12.704 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:13.073 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:13.073 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:13.074 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:13.075 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:13.078 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:13.079 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:13.080 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:13.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:13.082 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:13.083 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:18.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:18.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:18.085 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:18.128 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:18.128 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:18.498 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:18.499 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:18.499 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:18.499 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:18.499 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:18.499 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:18.500 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:18.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:18.500 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:18.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:18.500 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:18.500 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:18.501 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:18.501 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:18.501 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:18.502 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:18.502 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:18.502 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:18.502 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:18.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:18.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:18.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:18.503 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:18.504 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:18.504 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:18.504 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:18.504 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:18.505 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:18.505 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:18.505 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:18.505 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:18.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:18.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:18.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:18.508 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:18.508 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:18.508 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:18.508 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:18.509 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:18.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:18.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:18.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:18.510 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:18.510 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:18.510 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:18.511 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:18.511 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:18.511 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:18.511 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:18.512 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:18.512 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:18.512 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:18.512 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:18.513 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:18.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:18.513 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:18.513 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:18.514 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:18.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:18.514 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:18.515 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:23.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:23.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:23.518 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:23.562 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:23.563 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:23.922 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:23.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:23.924 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:23.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:23.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:23.927 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:23.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:23.929 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:23.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:23.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:23.931 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:23.931 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:23.931 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:23.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:23.931 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:23.932 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:28.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:28.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:28.934 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:28.974 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:28.975 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:29.336 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:29.336 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:29.337 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:29.338 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:29.339 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:29.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:29.341 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:29.342 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:29.343 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:29.344 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:29.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:29.345 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:29.346 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:34.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:34.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:34.348 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:34.386 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:34.387 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:34.757 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:35.134 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:35.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:40.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:40.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:40.138 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:40.182 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:40.182 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:40.543 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:40.544 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:40.544 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:40.544 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:40.544 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:40.545 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:40.546 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:40.547 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:40.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:40.549 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:40.550 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:40.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:40.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:40.553 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:40.553 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:45.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:45.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:45.556 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:45.598 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:45.598 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:45.973 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:45.974 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:45.974 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:45.974 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:45.974 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:45.975 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:45.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:45.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:45.979 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:45.980 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:45.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:45.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:45.983 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:45.984 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:50.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:50.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:50.986 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:51.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:51.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:51.398 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:51.398 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:51.398 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:51.399 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:51.399 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:51.399 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:51.399 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:51.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:51.400 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:51.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:51.400 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:51.400 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:51.400 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:51.400 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:51.401 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:51.401 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:51.401 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:51.401 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:51.402 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:51.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:51.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:51.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:51.402 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:51.402 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:51.403 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:51.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:51.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:51.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:51.404 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:51.406 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:51.406 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:51.406 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:51.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:51.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:51.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:51.407 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:51.407 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:51.407 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:51.407 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:51.408 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:51.408 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:51.408 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:51.408 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:51.408 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:51.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:51.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:51.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:51.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:51.409 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:51.410 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:51.410 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:51.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:51.410 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:51.411 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:16:56.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:56.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:16:56.414 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:16:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:16:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:16:56.825 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:16:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:16:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:16:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:16:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:16:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:16:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:16:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:16:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:16:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:16:56.835 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:16:56.835 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:01.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:01.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:01.838 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:01.880 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:01.881 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:02.249 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:02.250 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:02.251 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:02.254 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:02.259 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:02.260 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:07.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:07.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:07.262 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:07.302 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:07.302 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:07.571 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:07.571 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:07.572 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:07.573 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:07.574 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:07.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:07.576 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:07.577 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:07.578 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:07.579 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:07.580 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:07.580 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:07.581 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:12.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:12.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:12.584 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:12.627 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:12.627 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:12.999 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:13.005 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:13.005 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:13.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:13.005 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:13.005 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:13.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:13.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:13.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:13.006 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:13.006 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:13.006 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:13.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:13.008 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:13.008 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:13.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:13.008 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:13.008 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:13.008 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:13.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:13.009 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:13.010 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:18.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:18.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:18.013 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:18.054 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:18.054 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:18.426 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:18.435 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:18.437 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:23.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:23.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:23.440 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:23.482 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:23.482 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:23.844 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:23.844 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:23.844 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:23.844 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:23.845 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:23.845 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:23.845 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:23.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:23.845 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:23.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:23.846 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:23.846 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:23.846 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:23.846 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:23.846 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:23.846 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:23.847 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:23.848 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:23.849 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:23.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:23.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:23.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:23.849 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:23.849 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:23.851 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:23.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:23.853 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:23.854 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:28.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:28.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:28.857 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:28.907 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:28.908 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:29.282 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:29.283 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:29.283 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:29.283 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:29.283 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:29.283 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:29.283 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:29.284 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:29.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:29.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:29.286 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:29.286 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:29.286 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:29.286 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:29.286 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:29.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:29.288 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:29.289 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:29.289 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:29.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:29.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:29.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:29.290 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:29.290 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:29.290 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:29.290 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:29.290 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:29.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:29.292 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:29.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:34.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:34.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:34.296 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:34.336 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:34.336 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:34.705 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:34.714 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:34.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:39.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:39.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:39.717 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:39.760 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:39.760 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:40.129 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:40.130 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:40.130 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:40.130 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:40.130 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:40.130 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:40.131 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:40.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:40.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:40.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:40.133 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:40.133 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:40.133 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:40.133 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:40.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:40.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:40.136 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:40.137 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:40.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:40.139 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:40.140 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:45.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:45.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:45.143 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:45.549 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:45.550 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:45.550 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:45.550 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:45.550 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:45.550 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:45.551 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:45.552 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:45.552 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:45.552 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:45.552 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:45.552 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:45.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:45.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:45.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:45.553 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:45.553 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:45.554 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:45.555 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:45.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:45.557 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:45.558 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:45.559 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:45.559 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:45.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:45.559 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:45.560 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:50.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:50.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:50.562 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:50.603 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:50.604 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:50.978 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:50.979 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:50.979 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:50.979 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:50.979 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:50.979 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:50.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:50.980 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:50.981 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:50.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:50.981 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:50.981 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:50.981 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:50.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:50.982 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:50.985 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:50.985 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:50.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:50.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:50.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:50.985 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:50.986 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:50.987 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:50.988 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:50.988 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:50.988 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:50.989 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:17:55.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:55.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:17:55.991 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:17:56.032 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:17:56.032 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:17:56.401 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:17:56.402 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:17:56.402 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:17:56.402 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:17:56.402 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:17:56.402 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:17:56.402 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:56.403 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:56.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:56.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:17:56.406 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:56.407 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:17:56.408 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:56.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:17:56.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:17:56.410 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:17:56.410 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:17:56.410 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:17:56.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:17:56.411 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:17:56.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:01.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:01.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:01.415 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:01.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:01.457 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:01.832 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:01.832 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:01.832 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:01.833 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:01.833 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:01.833 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:01.833 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:01.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:01.833 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:01.834 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:01.835 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:01.836 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:01.836 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:01.836 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:01.836 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:01.836 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:01.836 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:01.841 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:01.842 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:06.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:06.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:06.845 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:06.888 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:06.888 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:07.260 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:07.260 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:07.260 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:07.260 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:07.261 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:07.261 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:07.261 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:07.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:07.261 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:07.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:07.262 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:07.262 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:07.262 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:07.262 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:07.263 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:07.263 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:07.263 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:07.263 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:07.263 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:07.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:07.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:07.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:07.264 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:07.264 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:07.264 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:07.264 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:07.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:07.266 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:07.267 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:07.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:07.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:07.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:07.269 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:07.269 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:07.269 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:07.269 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:07.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:07.271 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:07.273 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:12.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:12.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:12.275 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:12.314 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:12.314 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:12.684 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:12.685 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:12.685 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:12.685 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:12.686 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:12.686 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:12.686 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:12.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:12.686 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:12.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:12.687 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:12.687 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:12.687 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:12.687 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:12.688 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:12.688 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:12.688 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:12.688 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:12.688 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:12.689 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:12.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:12.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:12.689 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:12.689 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:12.690 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:12.690 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:12.690 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:12.690 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:12.690 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:12.691 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:12.691 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:12.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:12.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:12.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:12.692 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:12.692 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:12.692 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:12.692 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:12.693 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:12.693 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:12.693 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:12.693 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:12.693 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:12.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:12.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:12.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:12.694 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:12.695 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:12.695 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:12.695 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:12.695 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:12.698 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:12.699 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:17.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:17.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:17.703 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:17.746 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:17.746 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:18.123 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:18.137 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:18.139 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:23.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:23.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:23.142 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:23.184 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:23.184 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:23.554 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:23.563 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:23.564 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:28.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:28.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:28.567 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:28.608 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:28.608 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:28.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:28.982 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:28.983 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:28.984 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:28.985 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:28.986 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:28.986 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:28.986 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:28.987 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:33.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:33.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:33.990 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:34.033 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:34.400 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:34.420 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:34.422 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:39.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:39.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:39.425 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:39.466 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:39.467 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:39.839 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:39.850 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:39.852 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:44.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:44.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:44.855 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:44.899 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:44.899 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:45.269 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:45.279 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:45.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:50.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:50.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:50.284 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:50.328 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:50.694 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:50.695 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:50.695 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:50.695 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:50.695 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:50.695 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:50.695 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:50.696 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:50.697 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:50.697 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:50.697 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:50.697 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:50.697 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:50.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:50.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:50.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:50.698 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:50.698 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:50.698 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:50.698 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:50.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:50.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:50.700 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:50.700 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:50.700 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:50.700 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:50.700 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:50.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:50.702 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:50.703 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:50.704 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:50.704 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:50.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:50.704 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:50.705 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:18:55.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:55.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:18:55.708 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:18:55.749 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:18:55.750 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:18:56.124 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:18:56.125 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:18:56.125 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:18:56.125 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:18:56.125 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:18:56.125 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:18:56.125 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:56.126 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:56.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:56.128 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:56.129 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:56.130 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:18:56.131 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:56.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:18:56.133 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:18:56.134 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:01.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:01.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:01.137 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:01.179 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:01.179 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:01.546 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:01.547 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:01.547 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:01.547 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:01.547 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:01.549 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:01.549 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:01.549 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:01.549 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:01.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:01.549 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:01.550 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:01.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:01.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:01.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:01.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:01.552 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:01.552 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:01.552 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:01.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:01.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:01.556 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:01.557 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:06.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:06.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:06.559 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:06.600 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:06.601 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:06.971 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:06.972 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:06.972 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:06.972 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:06.972 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:06.973 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:06.974 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:06.975 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:06.977 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:06.978 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:06.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:06.980 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:06.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:06.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:06.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:11.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:11.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:11.984 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:12.402 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:12.411 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:12.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:17.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:17.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:17.414 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:17.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:17.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:17.823 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:17.834 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:17.835 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:22.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:22.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:22.838 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:22.880 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:22.880 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:23.248 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:23.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:23.250 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:23.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:23.252 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:23.252 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:23.252 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:23.252 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:23.252 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:23.252 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:23.254 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:23.254 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:23.254 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:23.254 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:23.254 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:23.255 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:23.256 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:23.257 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:23.258 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:23.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:23.258 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:23.259 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:28.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:28.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:28.261 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:28.302 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:28.302 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:28.676 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:28.677 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:28.677 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:28.678 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:28.678 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:28.678 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:28.678 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:28.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:28.679 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:28.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:28.679 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:28.680 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:28.680 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:28.680 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:28.680 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:28.680 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:28.681 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:28.681 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:28.681 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:28.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:28.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:28.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:28.682 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:28.682 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:28.682 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:28.682 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:28.683 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:28.683 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:28.683 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:28.683 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:28.683 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:28.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:28.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:28.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:28.684 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:28.684 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:28.685 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:28.685 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:28.685 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:28.685 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:28.685 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:28.685 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:28.686 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:28.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:28.688 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:28.689 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:33.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:33.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:33.692 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:33.731 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:33.732 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:34.101 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:34.102 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:34.102 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:34.102 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:34.102 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:34.102 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:34.102 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:34.103 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:34.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:34.105 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:34.106 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:34.108 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:34.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:34.111 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:34.112 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:39.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:39.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:39.114 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:39.156 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:39.157 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:39.535 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:39.535 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:39.536 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:39.536 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:39.536 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:39.536 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:39.537 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:39.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:39.537 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:39.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:39.538 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:39.538 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:39.539 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:39.539 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:39.539 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:39.539 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:39.540 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:39.540 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:39.540 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:39.541 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:39.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:39.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:39.541 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:39.542 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:39.542 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:39.542 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:39.542 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:39.543 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:39.543 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:39.543 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:39.543 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:39.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:39.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:39.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:39.545 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:39.545 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:39.545 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:39.545 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:39.546 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:39.546 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:39.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:39.547 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:39.547 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:39.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:39.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:39.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:39.548 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:39.548 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:39.549 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:39.549 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:39.549 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:39.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:39.550 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:39.550 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:39.550 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:39.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:39.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:39.551 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:39.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:39.552 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:39.552 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:39.552 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:39.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:39.553 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:39.554 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:44.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:44.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:44.556 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:44.598 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:44.598 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:44.970 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:44.970 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:44.970 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:44.971 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:44.971 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:44.971 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:44.971 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:44.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:44.972 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:44.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:44.972 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:44.972 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:44.972 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:44.972 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:44.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:44.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:44.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:44.976 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:44.976 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:44.976 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:44.976 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:44.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:44.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:44.978 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:44.978 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:44.978 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:44.978 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:44.978 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:44.979 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:44.979 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:44.979 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:44.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:44.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:44.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:44.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:44.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:44.980 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:44.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:44.980 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:44.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:44.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:44.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:49.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:49.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:49.984 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:50.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:50.028 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:50.401 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:50.401 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:50.401 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:50.401 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:50.401 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:50.402 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:50.402 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:50.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:50.402 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:50.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:50.403 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:50.403 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:50.403 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:50.403 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:50.403 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:50.403 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:50.404 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:50.405 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:50.405 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:50.405 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:50.405 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:50.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:50.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:50.406 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:50.406 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:50.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:50.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:50.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:50.407 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:50.407 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:50.407 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:50.408 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:50.408 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:50.408 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:50.408 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:50.408 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:50.409 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:50.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:50.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:50.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:50.409 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:50.409 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:50.410 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:50.410 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:50.410 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:50.410 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:50.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:50.412 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:50.412 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:50.412 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:50.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:50.412 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:50.413 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:19:55.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:55.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:19:55.416 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:19:55.459 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:19:55.460 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:19:55.838 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:19:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:19:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:19:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:19:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:19:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:19:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:19:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:19:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:19:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:19:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:19:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:19:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:19:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:19:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:19:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:19:55.848 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:19:55.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:00.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:00.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:00.851 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:00.893 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:00.894 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:01.267 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:01.268 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:01.268 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:01.268 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:01.268 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:01.268 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:01.269 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:01.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:01.271 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:01.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:01.273 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:01.273 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:01.273 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:01.273 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:01.273 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:01.273 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:01.274 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:01.274 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:01.274 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:01.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:01.275 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:01.276 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:01.276 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:01.276 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:01.276 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:01.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:01.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:01.277 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:01.278 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:06.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:06.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:06.281 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:06.321 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:06.321 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:06.685 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:06.686 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:06.686 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:06.686 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:06.686 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:06.686 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:06.687 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:06.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:06.689 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:06.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:06.691 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:06.692 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:06.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:06.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:06.694 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:06.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:06.694 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:06.694 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:06.694 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:06.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:06.695 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:06.696 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:11.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:11.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:11.698 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:11.738 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:11.738 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:12.113 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:12.120 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:12.123 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:12.124 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:17.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:17.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:17.126 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:17.167 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:17.167 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:17.542 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:17.552 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:17.553 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:22.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:22.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:22.556 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:22.597 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:22.597 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:22.967 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:22.967 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:22.967 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:22.967 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:22.969 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:22.970 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:22.970 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:22.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:22.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:22.970 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:22.970 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:22.971 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:22.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:22.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:22.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:22.972 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:22.972 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:22.972 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:22.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:22.974 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:22.975 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:22.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:22.976 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:22.976 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:22.976 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:22.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:22.976 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:22.977 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:27.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:27.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:27.980 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:28.022 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:28.023 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:28.391 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:28.392 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:28.392 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:28.393 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:28.393 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:28.393 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:28.394 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:28.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:28.394 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:28.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:28.395 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:28.395 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:28.395 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:28.396 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:28.396 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:28.396 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:28.397 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:28.397 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:28.397 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:28.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:28.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:28.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:28.398 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:28.399 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:28.399 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:28.399 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:28.399 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:28.400 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:28.400 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:28.400 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:28.401 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:28.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:28.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:28.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:28.402 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:28.402 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:28.402 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:28.403 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:28.403 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:28.403 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:28.404 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:28.404 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:28.404 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:28.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:28.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:28.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:28.405 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:28.406 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:28.406 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:28.406 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:28.406 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:28.407 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:28.407 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:28.407 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:28.408 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:28.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:28.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:28.408 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:28.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:28.409 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:28.409 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:28.410 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:28.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:28.410 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:28.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:33.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:33.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:33.414 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:33.827 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:33.828 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:33.828 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:33.828 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:33.828 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:33.828 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:33.829 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:33.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:33.829 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:33.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:33.830 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:33.830 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:33.830 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:33.830 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:33.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:33.832 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:33.833 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:33.834 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:33.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:33.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:33.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:33.835 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:33.835 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:33.835 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:33.836 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:33.836 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:33.836 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:33.836 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:33.836 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:33.836 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:33.837 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:33.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:33.838 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:33.839 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:38.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:38.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:38.841 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:38.882 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:38.882 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:39.262 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:39.271 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:39.272 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:44.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:44.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:44.275 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:44.311 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:44.311 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:44.679 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:44.679 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:44.680 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:44.681 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:44.682 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:44.683 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:44.684 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:44.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:44.687 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:44.688 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:44.688 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:44.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:44.688 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:44.689 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:49.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:49.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:49.691 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:49.732 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:49.733 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:50.202 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:50.202 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:50.206 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:50.207 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:50.208 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:50.209 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:50.210 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:50.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:50.211 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:50.211 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:50.211 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:50.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:50.211 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:50.212 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782128.213216_529.1 target=1_1602781307.714634_418.1 2020-10-15T17:20:55.214 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... FAIL 2020-10-15T17:20:55.215 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=iybjjr-69 2020-10-15T17:20:55.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:55.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:55.460 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:55.494 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:55.494 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:55.701 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T17:20:55.732 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:20:55.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:55.733 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:20:55.733 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:20:55.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:55.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:55.733 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602782455.255602_541.1", 2020-10-15T17:20:55.733 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:20:55.255602Z" 2020-10-15T17:20:55.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:55.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:55.734 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:20:55.734 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:20:55.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:55.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:55.735 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:20:55.735 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:20:55.735 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:55.735 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:20:55.736 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T17:20:55.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:55.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:55.738 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:56.022 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:56.022 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:56.023 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:56.023 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:56.023 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:56.024 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:56.024 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:56.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.024 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:56.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.025 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:56.025 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.026 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.026 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:56.026 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.026 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:56.027 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.027 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:56.027 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:56.028 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.028 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:56.029 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.029 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.029 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602782455.255602_541.1", 2020-10-15T17:20:56.029 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.030 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:56.030 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.030 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:20:55.255602Z", 2020-10-15T17:20:56.031 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:56.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.032 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:56.032 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.032 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.032 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:56.033 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.033 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:56.033 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.034 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:56.034 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:56.034 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.035 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:56.035 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.036 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.036 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:56.036 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.036 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:56.037 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.037 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:56.037 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:56.037 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:56.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.039 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:56.039 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:56.039 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:56.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.040 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:56.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T17:20:56.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T17:20:56.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:56.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:56.044 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:56.085 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:56.085 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:56.338 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:20:56.338 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:20:56.338 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:20:56.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:20:56.340 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:20:56.341 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602782455.255602_541.1", 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:20:55.255602Z", 2020-10-15T17:20:56.342 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:20:56.343 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:56.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:20:56.345 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:20:56.346 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:20:56.347 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:20:56.347 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:20:56.347 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:20:56.347 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:20:56.348 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T17:20:56.348 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T17:20:56.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:56.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:20:56.350 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:20:56.390 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:20:56.390 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:20:56.770 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:20:56.770 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:20:56.770 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:20:56.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:20:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:20:56.773 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:20:56.774 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.775 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:20:56.777 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:20:56.778 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:20:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:20:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:20:56.779 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:20:56.779 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:20:56.780 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:01.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:01.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:01.782 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:01.823 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:01.823 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:02.190 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:02.190 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:02.190 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:02.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:02.192 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:02.193 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:02.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:02.195 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:02.196 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:02.197 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:02.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:02.199 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:02.200 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:07.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:07.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:07.203 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:07.247 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:07.248 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:07.606 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:07.606 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:07.606 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:07.606 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:07.607 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:07.607 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:07.607 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:07.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:07.607 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:07.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:07.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:07.609 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:07.609 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:07.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:07.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:07.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:07.609 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:07.610 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:07.611 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:07.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:07.613 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:07.614 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:07.615 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:07.615 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:07.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:07.615 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:07.616 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:12.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:12.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:12.619 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:13.023 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:13.023 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:13.023 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:13.023 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:13.023 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:13.024 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:13.025 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:13.026 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:13.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:13.028 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:13.029 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:13.030 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:13.031 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:13.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:13.032 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:13.033 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:18.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:18.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:18.035 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:18.075 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:18.075 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:18.450 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:18.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:18.460 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:18.460 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:18.460 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:18.468 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:18.469 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:23.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:23.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:23.472 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:23.517 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:23.517 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:23.883 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:23.883 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:23.883 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:23.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:23.885 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:23.885 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:23.885 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:23.885 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:23.885 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:23.885 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:23.886 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:23.886 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:23.886 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:23.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:23.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:23.887 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:23.888 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:23.890 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:23.890 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:23.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:23.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:23.890 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:23.890 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:23.891 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:23.891 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:23.891 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:23.891 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:23.891 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:23.891 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:23.892 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:23.893 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:23.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:23.893 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:23.894 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:28.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:28.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:28.897 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:28.939 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:28.940 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:29.308 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:29.308 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:29.308 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:29.308 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:29.309 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:29.310 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:29.310 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:29.310 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:29.310 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:29.310 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:29.310 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:29.311 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:29.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:29.313 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:29.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:29.315 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:29.317 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:29.317 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:29.317 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:29.318 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:34.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:34.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:34.320 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:34.362 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:34.362 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:34.733 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:34.743 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:34.744 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:39.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:39.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:39.747 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:40.152 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:40.153 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:40.153 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:40.153 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:40.153 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:40.153 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:40.154 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:40.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:40.156 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:40.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:40.158 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:40.158 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:40.158 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:40.158 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:40.158 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:40.158 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:40.159 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:40.159 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:40.159 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:40.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:40.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:40.160 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:40.162 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:40.162 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:40.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:40.162 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:40.163 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:45.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:45.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:45.165 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:45.206 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:45.206 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:45.579 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:45.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:45.581 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:45.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:45.583 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:45.584 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:45.585 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:45.586 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:45.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:45.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:45.588 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:45.588 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:45.588 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:45.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:45.588 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:45.589 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:50.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:50.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:50.592 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:50.633 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:50.634 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:50.999 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:50.999 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:51.000 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:51.000 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:51.000 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:51.000 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:51.000 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:51.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:51.001 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:51.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:51.001 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:51.001 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:51.001 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:51.004 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:51.004 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:51.004 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:51.005 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:51.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:51.005 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:51.005 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:51.005 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:51.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:51.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:51.006 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:51.006 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:51.006 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:51.006 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:51.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:51.009 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:51.009 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:51.009 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:51.009 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:51.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:51.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:51.010 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:51.011 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:21:56.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:56.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:21:56.014 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:21:56.057 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:21:56.057 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:21:56.429 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:21:56.430 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:21:56.430 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:21:56.430 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:21:56.430 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:21:56.430 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:56.431 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:56.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:21:56.433 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:56.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:56.435 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:21:56.436 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:21:56.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:21:56.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:21:56.439 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:21:56.439 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:01.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:01.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:01.442 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:01.483 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:01.483 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:01.857 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:01.867 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:01.868 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:06.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:06.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:06.870 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:06.911 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:06.911 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:07.271 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:07.272 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:07.272 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:07.272 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:07.272 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:07.272 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:07.272 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:07.273 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:07.274 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:07.274 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:07.274 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:07.274 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:07.274 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:07.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:07.275 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:07.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:07.277 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:07.278 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:07.278 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:07.278 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:07.278 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:07.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:07.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:07.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:07.279 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:07.279 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:07.279 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:07.279 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:07.279 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:07.280 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:07.280 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:07.280 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:07.280 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:07.280 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:07.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:07.281 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:07.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:07.281 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:07.281 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:07.281 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:07.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:07.282 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:07.283 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:12.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:12.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:12.286 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:12.328 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:12.329 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:12.695 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:12.695 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:12.696 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:12.697 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:12.698 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:12.699 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:12.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:12.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:12.701 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:12.701 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:12.701 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:12.701 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:12.701 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:12.702 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:12.702 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:12.702 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:12.702 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:12.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:12.703 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:12.704 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:12.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:12.705 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:12.705 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:12.705 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:12.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:12.705 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:12.706 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:17.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:17.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:17.710 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:17.751 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:17.751 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:18.117 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:18.126 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:18.127 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:23.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:23.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:23.131 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:23.175 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:23.175 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:23.544 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:23.558 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:23.560 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:28.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:28.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:28.563 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:28.603 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:28.603 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:28.971 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:28.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:28.973 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:28.974 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:28.974 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:28.974 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:28.974 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:28.974 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:28.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:28.975 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:28.976 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:28.978 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:28.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:28.980 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:28.981 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:33.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:33.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:33.984 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:34.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:34.027 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:34.402 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:34.413 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:34.414 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:39.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:39.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:39.417 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:39.460 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:39.460 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:39.831 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:39.840 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:39.841 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:44.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:44.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:44.844 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:44.886 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:44.886 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:45.260 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:45.269 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:45.270 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:50.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:50.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:50.273 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:50.311 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:50.312 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:50.670 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:50.670 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:50.671 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:50.673 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:50.673 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:50.673 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:50.673 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:50.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:50.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:50.674 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:50.674 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:50.674 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:50.674 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:50.674 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:50.675 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:50.676 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:50.676 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:50.676 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:50.676 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:50.676 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:50.676 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:50.677 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:50.678 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:50.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:50.680 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:50.681 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:22:55.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:55.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:22:55.683 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:22:55.722 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:22:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:22:56.099 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:22:56.099 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:22:56.099 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:22:56.099 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:56.100 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:56.101 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:56.102 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:22:56.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:56.104 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:22:56.105 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:22:56.106 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:22:56.107 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:22:56.108 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:22:56.108 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:22:56.109 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:01.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:01.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:01.111 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:01.156 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:01.525 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:01.525 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:01.526 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:01.526 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:01.526 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:01.526 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:01.526 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:01.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:01.527 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:01.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:01.531 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:01.532 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:01.533 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:01.534 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:01.535 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:01.536 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:06.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:06.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:06.538 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:06.578 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:06.579 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:06.946 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:06.947 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:06.947 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:06.947 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:06.947 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:06.947 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:06.948 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:06.949 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:06.950 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:06.951 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:06.952 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:06.954 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:06.955 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:06.956 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:11.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:11.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:11.959 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:12.002 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:12.002 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:12.378 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:12.387 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:12.388 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:17.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:17.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:17.391 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:17.431 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:17.431 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:17.805 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:17.814 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:17.815 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:22.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:22.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:22.818 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:22.858 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:22.859 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:23.230 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:23.230 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:23.231 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:23.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:23.232 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:23.232 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:23.232 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:23.232 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:23.232 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:23.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:23.235 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:23.236 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:23.237 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:23.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:23.239 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:23.240 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:28.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:28.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:28.244 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:28.286 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:28.287 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:28.655 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:28.656 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:28.656 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:28.656 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:28.656 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:28.656 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:28.657 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:28.658 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:28.660 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:28.662 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:28.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:28.664 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:28.665 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:33.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:33.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:33.668 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:33.708 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:33.708 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:34.081 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:34.081 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:34.082 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:34.082 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:34.082 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:34.082 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:34.082 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:34.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:34.083 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:34.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:34.085 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:34.086 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:34.086 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:34.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:34.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:34.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:34.086 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:34.087 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:34.088 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:34.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:34.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:34.091 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:34.092 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:39.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:39.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:39.094 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:39.134 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:39.134 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:39.505 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:39.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:39.507 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:39.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:39.509 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:39.510 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:39.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:39.512 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:39.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:39.514 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:39.514 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:39.514 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:39.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:39.514 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:39.515 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:44.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:44.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:44.518 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:44.561 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:44.936 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:44.937 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:44.937 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:44.937 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:44.937 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:44.937 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:44.937 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:44.938 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:44.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:44.940 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:44.941 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:44.942 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:44.943 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:44.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:44.945 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:44.946 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:49.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:49.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:49.950 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:49.990 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:49.991 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:50.361 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:50.361 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:50.362 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:50.362 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:50.362 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:50.362 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:50.362 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:50.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:50.363 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:50.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:50.363 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:50.363 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:50.363 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:50.363 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:50.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:50.365 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:50.366 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:50.367 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:50.368 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:50.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:50.370 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:50.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:23:55.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:55.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:23:55.374 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:23:55.416 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:23:55.417 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:23:55.785 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:23:55.785 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:23:55.785 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:23:55.785 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:55.786 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:23:55.787 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:55.787 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:55.787 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:23:55.787 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:55.787 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:23:55.787 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:23:55.788 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:23:55.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:55.790 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:55.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:23:55.792 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:23:55.793 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:23:55.794 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:23:55.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:23:55.794 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:23:55.795 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:00.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:00.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:00.797 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:00.840 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:00.840 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:01.212 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:01.212 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:01.213 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:01.213 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:01.213 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:01.213 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:01.213 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:01.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:01.214 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:01.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:01.216 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:01.217 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:01.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:01.219 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:01.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:01.221 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:01.221 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:01.221 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:01.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:01.221 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:01.222 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:06.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:06.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:06.224 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:06.263 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:06.264 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:06.632 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:06.633 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:06.633 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:06.633 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:06.633 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:06.634 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:06.635 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:06.636 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:06.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:06.638 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:06.639 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:06.640 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:06.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:06.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:06.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:06.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:06.641 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:06.642 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:06.642 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:06.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:06.642 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:06.643 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:11.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:11.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:11.646 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:11.683 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:11.683 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:12.056 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:12.057 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:12.057 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:12.057 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:12.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:12.057 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:12.058 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:12.059 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:12.060 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:12.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:12.062 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:12.062 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:12.062 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:12.062 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:12.062 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:12.062 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:12.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:12.064 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:12.065 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:12.066 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:12.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:12.066 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:12.067 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:17.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:17.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:17.069 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:17.110 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:17.110 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:17.482 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:17.492 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:17.492 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:22.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:22.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:22.500 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:22.540 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:22.540 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:22.913 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:22.913 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:22.913 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:22.913 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:22.914 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:22.914 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:22.914 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:22.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:22.914 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:22.915 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:22.916 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:22.917 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:22.918 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:22.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:22.920 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:22.921 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:22.921 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:22.921 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:22.921 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:22.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:22.922 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:22.923 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:22.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:22.923 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:22.924 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:27.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:27.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:27.926 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:27.966 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:27.967 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:28.338 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:28.339 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:28.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:28.342 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:28.343 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:28.344 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:28.345 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:28.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:28.347 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:28.348 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:33.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:33.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:33.351 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:33.393 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:33.761 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:33.762 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:33.762 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:33.762 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:33.762 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:33.762 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:33.762 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:33.764 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:33.764 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:33.764 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:33.764 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:33.764 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:33.764 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:33.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:33.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:33.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:33.765 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:33.765 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:33.765 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:33.766 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:33.766 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:33.766 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:33.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:33.766 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:33.767 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:33.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:33.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:33.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:33.767 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:33.767 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:33.768 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:33.768 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:33.768 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:33.768 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:33.768 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:33.768 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:33.769 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:33.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:33.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:33.772 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:33.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:38.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:38.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:38.775 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:38.817 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:38.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:39.186 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:39.195 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:39.196 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:44.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:44.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:44.198 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:44.241 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:44.241 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:44.605 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:44.605 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:44.605 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:44.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:44.609 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:44.610 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:44.612 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:44.613 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:44.614 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:44.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:44.614 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:44.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:49.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:49.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:49.618 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:49.662 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:49.662 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:50.033 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:50.033 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:50.034 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:50.034 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:50.034 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:50.034 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:50.034 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:50.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:50.035 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:50.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:50.035 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:50.035 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:50.035 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:50.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:50.037 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:50.038 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:50.038 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:50.038 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:50.038 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:50.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:50.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:50.039 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:50.040 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:50.041 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:50.041 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:50.041 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:50.041 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:50.041 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:50.041 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:50.042 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:50.042 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:50.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:50.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:50.042 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:50.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:50.043 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:50.043 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:50.043 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:50.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:50.044 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:50.045 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:24:55.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:55.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:24:55.047 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:24:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:24:55.090 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:24:55.463 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:24:55.463 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:24:55.464 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:24:55.464 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:24:55.464 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:24:55.464 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:24:55.465 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:55.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:55.465 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:24:55.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:55.466 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:24:55.466 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:55.466 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:55.467 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:24:55.467 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:55.467 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:24:55.467 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:55.468 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:24:55.468 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:55.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:55.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:55.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:55.469 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:24:55.470 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:55.470 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:55.470 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:24:55.471 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:55.471 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:55.471 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:55.472 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:24:55.472 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:24:55.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:55.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:55.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:55.473 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:24:55.474 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:55.474 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:55.474 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:55.474 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:55.475 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:55.475 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:55.475 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:55.475 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:55.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:55.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:55.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:24:55.476 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:24:55.476 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:24:55.476 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:24:55.476 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:24:55.476 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:55.477 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:24:55.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:24:55.478 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:24:55.478 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:24:55.478 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:24:55.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:24:55.479 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:24:55.480 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:00.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:00.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:00.483 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:00.524 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:00.525 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:00.897 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:00.914 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:00.916 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:05.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:05.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:05.918 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:05.961 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:05.961 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:06.338 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:06.338 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:06.338 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:06.338 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:06.339 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:06.339 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:06.339 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:06.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:06.340 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:06.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:06.340 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:06.341 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:06.341 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:06.341 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:06.341 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:06.342 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:06.342 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:06.342 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:06.343 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:06.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:06.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:06.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:06.344 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:06.344 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:06.344 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:06.344 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:06.345 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:06.345 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:06.345 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:06.346 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:06.346 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:06.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:06.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:06.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:06.346 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:06.347 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:06.348 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:06.349 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:06.349 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:06.349 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:06.350 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:06.350 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:06.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:06.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:06.351 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:06.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:06.351 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:06.352 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:06.352 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:06.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:06.353 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:06.354 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:11.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:11.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:11.357 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:11.398 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:11.398 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:11.769 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:11.769 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:11.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:11.771 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:11.771 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:11.771 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:11.771 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:11.771 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:11.772 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:11.773 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:11.774 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:11.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:11.776 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:11.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:11.778 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:11.778 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:11.778 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:11.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:11.778 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:11.779 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:16.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:16.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:16.781 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:16.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:16.822 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:17.201 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:17.205 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:17.210 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:17.211 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:22.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:22.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:22.213 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:22.256 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:22.256 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:22.612 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:22.627 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:22.628 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:27.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:27.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:27.631 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:27.671 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:27.671 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:28.033 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:28.034 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:28.034 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:28.034 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:28.034 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:28.034 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:28.034 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:28.035 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:28.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:28.037 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:28.037 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:28.037 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:28.037 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:28.037 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:28.037 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:28.038 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:28.038 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:28.038 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:28.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:28.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:28.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:28.039 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:28.039 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:28.039 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:28.039 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:28.039 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:28.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:28.041 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:28.042 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:28.043 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:28.043 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:28.043 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:28.043 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:28.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:33.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:33.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:33.046 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:33.086 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:33.087 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:33.458 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:33.465 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:33.466 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:33.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:33.467 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:33.468 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:38.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:38.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:38.470 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:38.513 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:38.883 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:38.892 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:38.893 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:43.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:43.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:43.897 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:43.938 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:43.938 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:44.313 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:44.314 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:44.314 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:44.314 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:44.315 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:44.315 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:44.315 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:44.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:44.316 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:44.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:44.316 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:44.317 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:44.317 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:44.317 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:44.317 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:44.317 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:44.318 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:44.318 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:44.318 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:44.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:44.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:44.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:44.319 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:44.319 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:44.320 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:44.320 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:44.320 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:44.320 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:44.321 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:44.321 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:44.321 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:44.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:44.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:44.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:44.322 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:44.323 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:44.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:44.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:44.326 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:44.327 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:49.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:49.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:49.331 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:49.374 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:49.374 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:49.743 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:49.755 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:49.757 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:25:54.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:54.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:25:54.759 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:25:54.799 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:25:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:25:55.175 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:25:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:25:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:25:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:25:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:25:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:25:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:25:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:25:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:25:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:25:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:25:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:25:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:25:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:25:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:25:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:25:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:25:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:25:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:25:55.185 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:25:55.186 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:26:00.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:00.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:00.189 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:00.236 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:00.236 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:00.599 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:26:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:26:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:26:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:26:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:26:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:26:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:26:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:00.609 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:26:00.610 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:26:05.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:05.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:05.612 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:05.652 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:05.653 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:06.021 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:26:06.021 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:26:06.021 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:26:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:26:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:26:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:26:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:26:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:26:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:26:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:06.034 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:26:06.035 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:26:11.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:11.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:11.037 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:11.078 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:11.078 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:11.451 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:26:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:26:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:26:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:26:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:26:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:26:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:26:11.454 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:11.454 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:26:11.454 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:11.454 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:26:11.454 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:11.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:26:11.455 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:11.456 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:11.456 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:11.456 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:26:11.456 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:11.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:11.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:11.457 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:11.458 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:11.459 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:26:11.460 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:26:11.461 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:26:11.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:11.461 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:26:11.462 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:26:16.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:16.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:16.464 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:16.979 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:26:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:26:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:26:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:26:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:26:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:26:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:26:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:26:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602781307.714634_418.1", 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:01:47.714634Z", 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:26:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:26:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:26:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:16.988 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:26:16.989 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1602782455.255602_541.1 target=1_1602781307.714634_418.1 2020-10-15T17:26:21.991 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2020-10-15T17:26:22.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-10-15T17:26:22.032 INFO:tasks.rgw_multisite_tests: 2020-10-15T17:26:22.032 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-15T17:26:22.033 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2020-10-15T17:26:22.033 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-15T17:26:22.033 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-15T17:26:22.034 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-15T17:26:22.034 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-15T17:26:22.034 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 1129, in test_bucket_sync_enable_right_after_disable 2020-10-15T17:26:22.034 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-15T17:26:22.035 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-15T17:26:22.035 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-15T17:26:22.035 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-15T17:26:22.035 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-15T17:26:22.036 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-15T17:26:22.036 INFO:tasks.rgw_multisite_tests: 2020-10-15T17:26:22.036 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-15T17:26:22.036 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable_enable 2020-10-15T17:26:22.037 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-15T17:26:22.037 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-15T17:26:22.037 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-15T17:26:22.037 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-15T17:26:22.037 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 1163, in test_bucket_sync_disable_enable 2020-10-15T17:26:22.038 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-15T17:26:22.038 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-15T17:26:22.038 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-15T17:26:22.038 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-15T17:26:22.039 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-15T17:26:22.039 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-15T17:26:22.039 INFO:tasks.rgw_multisite_tests: 2020-10-15T17:26:22.039 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-15T17:26:22.040 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_encrypted_object_sync 2020-10-15T17:26:22.040 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-15T17:26:22.040 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-15T17:26:22.040 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-15T17:26:22.041 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-15T17:26:22.041 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 1243, in test_encrypted_object_sync 2020-10-15T17:26:22.041 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-15T17:26:22.041 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-15T17:26:22.042 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-15T17:26:22.042 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-15T17:26:22.042 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-15T17:26:22.042 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-15T17:26:22.042 INFO:tasks.rgw_multisite_tests: 2020-10-15T17:26:22.043 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-15T17:26:22.043 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_index_log_trim 2020-10-15T17:26:22.043 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-15T17:26:22.043 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-15T17:26:22.044 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-15T17:26:22.044 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-15T17:26:22.044 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 1273, in test_bucket_index_log_trim 2020-10-15T17:26:22.044 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2020-10-15T17:26:22.045 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 1268, in make_test_bucket 2020-10-15T17:26:22.045 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-15T17:26:22.045 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-15T17:26:22.045 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-15T17:26:22.046 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuval-cls-fifo/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-15T17:26:22.046 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-15T17:26:22.046 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-15T17:26:22.046 INFO:tasks.rgw_multisite_tests: 2020-10-15T17:26:22.047 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-15T17:26:22.047 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 2640.327s 2020-10-15T17:26:22.047 INFO:tasks.rgw_multisite_tests: 2020-10-15T17:26:22.047 INFO:tasks.rgw_multisite_tests:FAILED (failures=4) 2020-10-15T17:26:22.050 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-10-15T17:26:22.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:22.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:22.053 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:22.093 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:22.093 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:22.323 INFO:teuthology.orchestra.run.smithi183.stderr:No --period given, using current period=69fb12ff-ae85-466c-bb2d-3018777b1732 2020-10-15T17:26:22.359 INFO:teuthology.orchestra.run.smithi183.stdout:[ 2020-10-15T17:26:22.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:22.359 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:26:22.359 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602782455.255602_541.1", 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "2020-10-15T17:20:55.255602Z" 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:22.360 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: "last_update": "0.000000" 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:22.361 INFO:teuthology.orchestra.run.smithi183.stdout:] 2020-10-15T17:26:22.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-15T17:26:22.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:22.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:22.364 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:22.403 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:22.403 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:22.694 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2020-10-15T17:26:22.694 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_status": { 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: "info": { 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "sync", 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: "num_shards": 4, 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:22.695 INFO:teuthology.orchestra.run.smithi038.stdout: "markers": [ 2020-10-15T17:26:22.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:22.696 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 0, 2020-10-15T17:26:22.696 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:22.696 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:22.696 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:26:22.697 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:22.697 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 1, 2020-10-15T17:26:22.697 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:22.697 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:26:22.697 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 1, 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:22.698 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "1_1602782455.255602_541.1", 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "2020-10-15T17:20:55.255602Z", 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 4 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:22.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 2, 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:22.700 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: "key": 3, 2020-10-15T17:26:22.701 INFO:teuthology.orchestra.run.smithi038.stdout: "val": { 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "state": 1, 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "marker": "", 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "next_step_marker": "", 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "total_entries": 0, 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "pos": 0, 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "timestamp": "0.000000", 2020-10-15T17:26:22.702 INFO:teuthology.orchestra.run.smithi038.stdout: "realm_epoch": 2 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: "full_sync": { 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: "total": 1, 2020-10-15T17:26:22.703 INFO:teuthology.orchestra.run.smithi038.stdout: "complete": 1 2020-10-15T17:26:22.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2020-10-15T17:26:22.704 INFO:teuthology.orchestra.run.smithi038.stdout:} 2020-10-15T17:26:22.705 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-15T17:26:22.705 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-15T17:26:22.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:22.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:22.706 INFO:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:22.747 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:22.747 INFO:teuthology.orchestra.run.smithi183.stderr:ignoring --setgroup ceph since I am not root 2020-10-15T17:26:23.015 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2020-10-15T17:26:23.016 INFO:teuthology.orchestra.run.smithi183.stdout: "sync_status": { 2020-10-15T17:26:23.016 INFO:teuthology.orchestra.run.smithi183.stdout: "info": { 2020-10-15T17:26:23.016 INFO:teuthology.orchestra.run.smithi183.stdout: "status": "sync", 2020-10-15T17:26:23.016 INFO:teuthology.orchestra.run.smithi183.stdout: "num_shards": 4, 2020-10-15T17:26:23.016 INFO:teuthology.orchestra.run.smithi183.stdout: "period": "69fb12ff-ae85-466c-bb2d-3018777b1732", 2020-10-15T17:26:23.016 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:26:23.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:23.017 INFO:teuthology.orchestra.run.smithi183.stdout: "markers": [ 2020-10-15T17:26:23.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:23.017 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 0, 2020-10-15T17:26:23.017 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:26:23.017 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:26:23.018 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602780154.156484_109.1", 2020-10-15T17:26:23.018 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:26:23.018 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 1, 2020-10-15T17:26:23.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:26:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T16:42:34.156484Z", 2020-10-15T17:26:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:26:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:23.019 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 1, 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "1_1602782455.255602_541.1", 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:26:23.020 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "2020-10-15T17:20:55.255602Z", 2020-10-15T17:26:23.021 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 4 2020-10-15T17:26:23.021 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:23.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:23.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:23.021 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 2, 2020-10-15T17:26:23.021 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:26:23.022 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:26:23.022 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:26:23.022 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:26:23.022 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:26:23.022 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:26:23.022 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:26:23.023 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:26:23.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:23.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:23.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2020-10-15T17:26:23.023 INFO:teuthology.orchestra.run.smithi183.stdout: "key": 3, 2020-10-15T17:26:23.023 INFO:teuthology.orchestra.run.smithi183.stdout: "val": { 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "state": 1, 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "marker": "", 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "next_step_marker": "", 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "total_entries": 0, 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pos": 0, 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "timestamp": "0.000000", 2020-10-15T17:26:23.024 INFO:teuthology.orchestra.run.smithi183.stdout: "realm_epoch": 2 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: "full_sync": { 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: "total": 1, 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: "complete": 1 2020-10-15T17:26:23.025 INFO:teuthology.orchestra.run.smithi183.stdout: } 2020-10-15T17:26:23.026 INFO:teuthology.orchestra.run.smithi183.stdout:} 2020-10-15T17:26:23.026 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-15T17:26:23.027 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-15T17:26:23.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:23.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-15T17:26:23.028 INFO:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-15T17:26:23.069 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setuser ceph since I am not root 2020-10-15T17:26:23.069 INFO:teuthology.orchestra.run.smithi038.stderr:ignoring --setgroup ceph since I am not root 2020-10-16T04:16:26.939 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-10-16T04:16:27.002 DEBUG:teuthology.task.console_log:Killing console logger for smithi183 2020-10-16T04:16:27.003 DEBUG:teuthology.task.console_log:Killing console logger for smithi038 2020-10-16T04:16:27.004 DEBUG:teuthology.task.console_log:Killing console logger for smithi183 2020-10-16T04:16:27.005 DEBUG:teuthology.task.console_log:Killing console logger for smithi038 2020-10-16T04:16:27.006 DEBUG:teuthology.exit:Finished running handlers